replacing flat-file db with proper database with record level editing

Question

I cannot install SQLite on a remote machine, so I have to find a way to store a large amount of data in some kind of database structure.

Example data

key,values...
key,values....
..

There are currently about a million rows in a 20MB flat file, and hourly I have to read through each record and value in the file and update or add a record. Since it is a flat file I have to rewrite the whole file each time.

I am looking at the Storable module, but I think it also writes data sequentially. I want to edit only those records which need to be changed.

reading and updating of random records is a requirement. Additions can be anywhere(order is not important)

Can anyone suggest something? How will I know if I can setup a native Berkeley database file on these systems, which are a mixture of Solaris and Linux?

________________finally__________________

finally I understood things better (thank you all), and based on your suggestions I used AnyDBM_File. It found NDBM_File ('C' library) installed on all OS. So far so good.

Just to check how it will play out in real world. I ran a sample script to add 1 million records (the max records i think i may ever get in a day, normally between 500k to 700k). OMG it created a 110G data file on my disk !!!! and all the records were like:

a628234 = 0.178532683639599

I mean my real world records are longer than that. compare this to a flat file which is holding real-life 700k+ records and is only 15Mb on disk.

I am disappointed with the slowness and bloat-ness of this, so for now i think i will pay the price by writing the whole file each time an edit is required.

Thanks again for all your help.

Does your flat file have a fixed record length, or is it comma-separated or similar? — Borodin
– Borodin, Commented Mar 27, 2015 at 22:57
record length is not fixed and that value changes, depending on program logic. Also if i use these various perl modules AnyDBM_File or DB_File. do i need a db server also, on my OS? — Rajeev
– Rajeev, Commented Mar 27, 2015 at 23:06
perldoc perlport has this to say about DBM: "There is no one DBM module available on all platforms. SDBM_File and the others are generally available on all Unix and DOSish ports, but not in MacPerl, where only NBDM_File and DB_File are available. The good news is that at least some DBM module should be available, and AnyDBM_File will use whichever module it can find. Of course, then the code needs to be fairly strict, dropping to the greatest common factor (e.g., not exceeding 1K for each record), so that it will work with any DBM module. See AnyDBM_File for more details." — ThisSuitIsBlackNot
– ThisSuitIsBlackNot, Commented Mar 27, 2015 at 23:13
ThisSuitIsBlackNot, thank you. but is the documentation talking about perl modules or db libraris installed in OS. in my situation i looked around and found: db.h on one server and dbm.h in another. So I am understanding that all these perl modules are just 'connectors' to the actual db libraries which are to be installed separately in the OS ? is that right? — Rajeev
– Rajeev, Commented Mar 27, 2015 at 23:28

edem · Accepted Answer · 2015-03-28 14:14:53Z

0

As they said in the comments you may use SDBM_File module. For example:

#!/usr/bin/perl 
use strict;
use warnings;
use v5.14;

use Fcntl;
use SDBM_File;

my $filename = "dbdb";

my %h;

tie %h, 'SDBM_File', $filename, O_RDWR|O_CREAT, 0666
    or die "Error: $!\n";

# To run only one time to fill the dbdb file.
# Next time you may delete this line and
# the output will be the same "16,40".    
$h{$_} = $_ * 2 . "," . $_ * 5  for 1..100;

say $h{8};

untie %h;

Output: 16,40

edited Mar 28, 2015 at 14:14

answered Mar 28, 2015 at 14:08

edem

3,2923 gold badges23 silver badges47 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

w.k · Accepted Answer · 2015-03-27 22:41:14Z

-1

Depends, what your program logic needs, but one solution is to partition database, based on keys. So you can deal with many smaller files instead of one big file.

answered Mar 27, 2015 at 22:41

w.k

8,4765 gold badges36 silver badges64 bronze badges

Collectives™ on Stack Overflow

replacing flat-file db with proper database with record level editing

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related