Data Engineering

396 readers

1 users here now

A community for discussion about data engineering

Icon base by Delapouite under CC BY 3.0 with modifications to add a gradient

founded 2 years ago

MODERATORS

[email protected]

Data Newbie Looking for Advice (self.data_engineering)

submitted 11 months ago by Pyr_Pressure to c/[email protected]

4 comments fedilink hide all child comments

Hello,

I am looking for some advice to help me out at my job. Apologies if this is the wrong place to ask.

So, basically my boss is a complete technophobe and all of our data is stored across multiple excel files in drop box and I'm looking for a way to change that into a centralized database. I know my way around a computer but writing code is not something I have ever been able to grasp well.

The main issue with our situation is that our workers are all completely remote, and no I don't mean work from home in the suburbs from a home office. They use little laptops with no data connection and go out gathering data every day from a variety of locations, sometimes not even cell coverage.

We need up to 20 people entering data all day long and then updating a centralized database at the end of the day when they get back home and have internet connection. It will generally all be new entries, no one will need to be updating old entries.

It would be nice to have some sort of data entry form in drop box and a centralized database on our local server at head office which pulls the data at the end of each day. Field workers would also need access to certain data such as addresses, contact info, maps, photos, historical data, etc. but not all of it. For example the worker in City A only needs access to the historical data from records in and around City A, and workers in City B only need access to records involving City B.

Is there any recommended options for software which can achieve this? It needs to be relatively user friendly and simple as our workers are typically biology oriented summer students, not programmers.

top 4 comments

sorted by: hot top controversial new old

[–] [email protected] 6 points 11 months ago (1 children)

First, if people are winging it, your first problem is going to be standardization and sterilization of the data. If everyone is using excel, for example, they should be filling in the same template which enforces data integrity (eg required fields and allowed values). Don’t allow anyone to roll their own. Standardize everything. At that point your job becomes pretty easy with solutions ranging from converting the data to CSVs or just uploading it to a sql database (eg mysql) with some gui.

I would recommend thinking pretty hard about whether there is a business value in limiting access to data that people “don’t need.” While it does make sense for reporting to have simplified views, managing user level access to specific subsets of data is often not justified for the level of effort required. If you’re dealing with actual (eg government) classified information or if there are highly valuable trade secrets that need protecting, that’s another story. But if it’s just “Jeff in NY doesn’t need to see Jane LA’s sales numbers,” it’s probably a waste of effort and will be a time and money sink. I’m saying that as someone who has worked on very highly classified systems.

Anyway, everything you’re talking about is a solved problem That can be solved using excel and maybe an off the shelf database that an intern could set up for you.

[–] Pyr_Pressure 3 points 11 months ago (1 children)

Thanks for the Quick Reply. Personally I don't have much of a need to deny access to the data but the boss gets nervous about seasonal employees getting access, in the past we had one employee copy the files they did have access to and try and start a competing business the following year and bid against us.

[–] [email protected] 2 points 11 months ago (1 children)

Fair enough, and after all it’s not your money.

In any case, it should be pretty cheap to have someone set it up for you. I’d throw it out there as somewhere around $30-50/hour with remote work allowed, or a fixed price of $1000-1500. From your description it sounds like something that could be knocked out in a week by someone with a few years experience, and you have the additional security of having someone else on the hook for, well, security. Just make sure they document everything.

[–] Pyr_Pressure 2 points 11 months ago

Awesome, thanks for the info