Big organisations are fantastic for new and aspiring data scientist. I would highly recommend those new to data science find a position in a company that already has a data science team. This will give the new data scientist team members to bounce ideas off and hopefully a mentor. Established organisations often have more realistic expectations of what you are capable of, well established procedures and clear business plans that enable a new data scientist to settle into their new roll without too much stress.
Start-ups on the other hand generally aren't that great for new data scientists. They can often expect too much of one person and if that person does not have the experience or skills required they can be chewed up and spat out quickly and harshly. Having said that, as an experienced data scientist, I love the challenges you face with start-ups!
Why I love start-ups:
Less staff so your role can be greatly varied. I don't just do data science/analytics, I also build dashboards, pull together databases and help design and create the front-end. I help out every department - marketing, sales and design as I am needed. In bigger established organisations your role is often very specific and no one steps outside of their narrowly scoped role. Ever.
Start-ups have less bureaucracy and anyone with an idea can be heard. Ideas can be tested and then implemented quickly. This is a HUGE advantage over big business who can take months to decide on the smallest of decisions that will have little to no impact at all in the long run. Start-ups can pivot and leap as required, bounding ahead while big business plod along day after day often in circles never changing, never getting anywhere.
Often with start-ups there are big wins we, as data scientists can achieve for the business quickly. Actually there are big wins in big organisations too, but due to the above points they aren't quick or easy to achieve.
Start-ups are fun, fresh and high energy and the people working there believe in what they are doing. You might need to learn new skills and/or emerging technology to help the start-up out. For most staff of big organisations you turn up, do your allocated work/role and then you head home. Essentially just cranking the same handle day after day, week after week. Everyone gets comfortable, but everything starts to feel stale, and while technology and techniques may advance and change the processes stay the same because "this is how we have always done it".
Currently, at least here in Australia the natural progression for start-ups has been:
1) get a bunch of people together and create a business plan
2) get funding then hire people to build a site
3) collect data as part of a natural business process
4) run the business by gut feel/intuition to a point
5) hire data analysts/scientists to create a more scientific approach to the optimization of the business
In the future, start-ups will have the algorithms/data science functionality from the start. This is how I think the start-up world should operate:
1) A couple of buddies get together and build a site and collect data with the aim of building a ML algorithm. If they don't have the skills, they learn them. That's right. They go and learn what they need to know, at least enough to get it working.
2) The data and algorithm is king. The business asset is the data.
3) The business is bootstrapped, there is no funding required.
Why do I feel this is the right way to do business?
Data is the ultimate authority. Really it doesn't matter what my opinion is, or your's, the CEO, CFO or anyone else. If there is evidence in the data that hypothesis A is correct (assuming we have used correct data, thought and skill) then hypothesis A is what we should go with.
Businesses need to have a data driven culture, and collect and use data right from the start to make informed business decisions.