[thenson]

Most of my programming projects that I have had to make in school dealt with mock programs that pretty much only used RAM. One example was a program that simulated a book store where the user was prompted about book title and other information and hit add where I made a linked list with many methods on how to retrieve/delete books etc.

Well obviously when the program was done, all the list information is gone due to thats how RAM works. What do professional programmers use to store information for long-term usage. I have written other programs that created a file that kept track of stuff using filereaders/writers that created files. I know this will work but is this really how professional programmers write?

I work for a security company and we have a program that our dispatch uses that logs the times people call, names of who responded and there is an area to write text like "possible shop lifting at guess store". I just dont see this professional program writing all of the information into a text file and being reread to the screen in GUI fields via an IO file stream.

I tried finding where the information was being stored but couldn't find anything.

I know that data-bases can be used its just maybe I am over thinking something with this.

So, what do professional programmers do with information that needs to be stored and saved for long-term usage?

[Chaosmaster]

Any information that needs to be stored for long-term usage, needs to be saved in a file.

There is different approaches how to do that. (which likely depends on how critical the information are)

It could be a proprietary binary format that might even use encryption or it could just be a simple xml-file.

[lolage]

I'd use a database personally, as well .. that's what databases are for, right?

Flatfile storage gets rather unweildy with a large data-set. With a database I can just fire off a query and get the data im looking for back pretty quickly. Now, you could of course memorymap the file but that gets a little fiddley as the size of the dataset increases.

Database is the best option as far as I can see.

[thenson]

lolage, on Jan 2 2009, 10:17 PM, said:

I'd use a database personally, as well .. that's what databases are for, right?

Flatfile storage gets rather unweildy with a large data-set. With a database I can just fire off a query and get the data im looking for back pretty quickly. Now, you could of course memorymap the file but that gets a little fiddley as the size of the dataset increases.

Database is the best option as far as I can see.

Yeah I understand how file writers and openers work for text files and have kept track of like 100 books that way that you could close programs and come back later and it will all be there..

The thing is that if you write a program like this is that the text file can be accessed by someone like myself who snoop around on the computer and doing so can easily allow it to be tampered with.

I have a fair understanding of how databases work, I just again seem to be reluctant to believe that programmers use the filewriters to store data which can easily be found and accessed.

[nutcracker]

thenson, on Jan 3 2009, 04:44 AM, said:

...
The thing is that if you write a program like this is that the text file can be accessed by someone like myself who snoop around on the computer and doing so can easily allow it to be tampered with.
...

If you're doing your software for Linux/Unix environment you can secure your files with proper reading and writing permissions. Then you can try to make your application be the only one to be able to access it for reading / writing / whatever. This way brings out the possibilities of different attacks on your software to reveal the contents of the forbidden files. I have never done this approach, so I can't give any more clues except that in the case you're using text files, you'd be better off using some sort of structured file format, like XML.

Not that it matters much, because I also agree the database is the way to go. If you go with SQL, in most database implementations you can set up different users, user roles and permissions. That way you will have a fast and secure way of storing your data.

[DToX]

EDIT: *removed something irrelevant*

suitability of files

For chronological data they may well be using files. Files are quite suited provided that logfile rotation is in use... it really doesn't matter whether the file is text-based or uses binary records, since the data is clearly chronological.

What sorting capabilities does it have/need, this is a big clue. If you can sort on many fields then text files are less likely. If, however, the application accesses the data sequentially or with with simple keywords rather than linkages between related records then a file would be a decent option.


To find out about the existing software, take a look at the file handles open on your system whilst using the program. This will reveal all ; )

There's certainly nothing unprofessional about using a file for a chronological record store with limited searches as appending is what they're good at, and their streaming nature tends to suit the usual access requirements for such data.


SECURITY

Obviously, databases allow for it and with files you have to define your own. However, databases may not be your most efficient route for chronological data.

Lets leave databases aside for the moment as securing them is covered at length elsewhere. With files you must rely on two things... encryption and access control. Encryption is a tricky monkey but access control is relatively simple.

Filesystem Access Controls

For access control it would be preferable to make your file restricted to either root, system or its own specific group for log R/W. Having said that, this helps you none if your application runs in the same security context, so you need a middleman. That middleman is typically run as a service/daemon and provides an IPC to the end user application which can now run in a standard users security context as a service consumer.

Now, I'm not a big fan of using system/root for daemons. So, let's create a new security context like 'my_seclogd' for our file permissions and daemon. Also, we don't want an exploitable daemon to spread damage any further so let's not grant it access to 'anything' other than what it needs... itself, its files, and possibly the temp store (or better, it's OWN temp store)

Now, the daemon can verify users permissions using a user file, lets say we split users into :

- Those that can add/create/modify users and permissions as well as perform logfile management.
- Those that can read/append (notice, append only... no overwriting in security logs, once written it's written)
- Those that can only read

Now you can connect to the daemons IPC mechanism, authenticate, and access the file through an append/read/seek API (whilst managers have additional read-user/write-user and some log management commands) as well as read-only to the log of operations (which shows an indelible record of what THEY did and which is cycled automatically by the daemon)

Okay... that's pretty robust.


Encryption

Encryption you can work out for yourself. Technically it adds surprisingly little protection on a local system (far less than disk encryption) so you need to work out where you want it, this will dictate where the key resides... either with the staff, on a file only accessible by system which starts the daemon or between the two. I'd tend towards 'between the two'

Staff starting the application could enter a key to prevent the filesystem holding all of the key, the daemon could be started at system start with a 'site key' from system to ensure the staff don't have the full key, and the daemon itself could incorporate a build key. That's three sources of key entropy :

- A staff key means that if the system is stolen by an outsider the data is secure.
- An inbuilt daemon key allows for differences among deployments such that logfiles from different sites are not transportable. This is the 'build' key.
- A key at daemon startup is the permanent 'site' or 'company' key.

All three keys brought together through daemon start and staff login to be merged (SHA would do) to reproduce the actual operating key from multiple sources. This key can then be expanded/condensed for use if required and applied to file I/O.

Obviously, it can be much safer to simply move the daemon offsite and have staff communicate with it using transactions over a secure link but the costs involved can be prohibitive and you introduce availability issues should the link drop (so, you still need a tamper-proof local cache) but it does offer protection against system seizure by a third party (read, theft)



I'd stress the importance of an append-only API.

Little story...

Back in the day we used to use a golfball printer for intrusion reports. This noisy little beast was locked in a cupboard (more for our sanity than for security) where it hammered out the security events related to both the building and the mainframe onto boxes of fanfold continuous stationary. The idea was... an attacker could erase the tapes in the offline store but the hardcopy was untouchable. Golfballs were used because the DM's were too complex for security and were unreadable without the ribbon... whilst this aggressive little bitch would almost cut shapes out of the paper if the ribbon wasn't present.

Then, one nightshift whilst playing around with the spare printers published codes (trying to triple-strike certain events to make them more visible) we discovered that although the printer did not allow reverse feed (reversing the paper) you could drop the printer into an undocumented mode which did allow reverse feeding by apparently tweaking the an internal feed size register to be larger than acceptable. So, one weekend we set up a second printer and simulated an attack...

... at the end of the attack which spawned about three pages of report on wide media, we multistriked each line with 5 overprints of some carefully chosen letters aimed at obscuring the log, then reverse feed and repeat. We ran the script and completely destroyed details of the simulated attack in the hardcopy.

The result was we got time off to play with the ROM and came up with a drop-in PROM which didn't allow reverse feeding at all. Not that such an attack was at all likely, but hey... time off is time off, and any distraction is good when you're spending your nights babysitting someones system and praying for a condition alarm ; )

Some later systems actually came WITH fully-featured epson line printers for security logging as a standard feature. Suffice to say, all too often your first sign of an attack was a few hundred pages of black sticky paper folded neatly back into the box it came from. XD


==DToX

This post has been edited by DToX: 03 January 2009 - 05:04 PM