2

As we process our csv data, we generate a lot of output files with 30 000 lines in each one of them. They all have the same columns/fields. They are all also in csv format and we put them into the same folder on the Linux server. The files are uniquely named using a combination of date, time and numeric digits. See below.

AB_20151127_120000_0_SEGMENT_FINAL.csv
AB_20151127_120000_1_SEGMENT_FINAL.csv
AB_20151127_120000_2_SEGMENT_FINAL.csv
AB_20151127_120000_3_SEGMENT_FINAL.csv
.
.
.
AB_20151127_120000_599_SEGMENT_FINAL.csv

So now we need to merge/join all of them into one big file called: AB_20151127_120000_SEGMENT_FINAL.csv (note the missing numeric digits from the merged file)

I tried awk as below but it is not working. Please tell me what I did wrong.

awk '"AB_20151127_120000_" NR-1 "_SEGMENT_FINAL.csv"' > AB_20151127_120000_SEGMENT_FINAL.csv
muru
  • 207,970
Gert Myburgh
  • 23
  • Are you looking for cat AB_20151127_120000_{1..599}_SEGMENT_FINAL.csv > AB_20151127_120000_SEGMENT_FINAL.csv? – muru Dec 09 '15 at 10:48
  • Thank you @muru. Let me try that quickly. Do I have to specify {1..599} or can I do something like {*} to say however many files there might be? – Gert Myburgh Dec 09 '15 at 10:55
  • If you use AB_20151127_120000_*_SEGMENT_FINAL.csv, the files will be in lexicographic order (1, 11, 12, ..., 199, 2, 21, ...). – muru Dec 09 '15 at 10:57
  • O I see. Your suggestion to use 'cat' is working. Thank you @muru. I just now need to know how many files gets generated when I originally split the files using 'awk'. This is all going to happen on the fly and I will have to work something into my script to see how many files were created. I can't just look at the file size and divide by 30 000 because although all of them have the same columns, the data size in those columns might differ from one file to the other. – Gert Myburgh Dec 09 '15 at 11:08
  • replace 599 with $(ls AB_20151127_120000_*.csv | wc -l) – Ralph Rönnquist Dec 09 '15 at 11:14
  • @GertMyburgh please [edit] your question and include everything you need. Tell us whether the order in which the files are joined is important, tell us that you also need to include the number of original files, explain where that should be included etc. – terdon Dec 09 '15 at 11:26
  • Your awk command has error. Instead of printing all the files like you wanted, and redirecting to a file, you were basically telling awk to look for a pattern "AB_20151127_120000_" NR-1 "_SEGMENT_FINAL.csv". Realy should be just awk AB_20151127_120000_*_SEGMENT_FINAL.csv '1' > outputFile.cvs (note 1 evaluates to true on each line, so it always prints). But cat is the way to go, however. You are not doing any special processing with the files, only creating one big file – Sergiy Kolodyazhnyy Dec 09 '15 at 12:12
  • @terdon yeah, i edited that already. And yeah, like I said cat is the way to go in this case. OP isn't doing any special processing with the files, just concatenating – Sergiy Kolodyazhnyy Dec 09 '15 at 12:19

2 Answers2

3

If the order in which the files are concatenated is not important, use:

cat AB_20151127_120000_*_SEGMENT_FINAL.csv > AB_20151127_120000_SEGMENT_FINAL.csv

If the order is important, you'll have to get creative. If you know the number of segments, 599 for example, you can use brace expansion (the \ is only there to let me print the command on two lines for readability):

cat AB_20151127_120000_{0..599}_SEGMENT_FINAL.csv > \
    AB_20151127_120000_SEGMENT_FINAL.csv

If you don't, you can still use brace expansion. Just choose a large enough number to be sure that all files will be included and ignore error messages about non-existant files:

cat AB_20151127_120000_{0..599}_SEGMENT_FINAL.csv > \
    AB_20151127_120000_SEGMENT_FINAL.csv 2>/dev/null

Alternatively, you can generate a list of sorted file names and use that:

cat $(printf '%s\n' AB_20151127_120000_*_SEGMENT_FINAL.csv | sort -nt_ -k4) > \
    AB_20151127_120000_SEGMENT_FINAL.csv

The printf will print each file name followed by a newline which is the passed to sort which will sort it numerically (-n) on the 4th field (-t4) where fields are defined by _ (-t_).

kos
  • 41,378
terdon
  • 104,404
0

If you have access to a Zsh shell, the task can be reduced to a single command:

cat AB_20151127_120000_*(n)_SEGMENT_FINAL.csv >AB_20151127_120000_SEGMENT_FINAL.csv

This is because the (n) globbing qualifier forces the * globbing pattern to expand to a list of filenames sorted in their natural order, as opposed to their lexicographical order.

For comparison, filename expansion in Bash:

$ for f in *; do echo "$f"; done
AB_20151127_120000_0_SEGMENT_FINAL.csv
AB_20151127_120000_10_SEGMENT_FINAL.csv
AB_20151127_120000_1_SEGMENT_FINAL.csv
AB_20151127_120000_2_SEGMENT_FINAL.csv
AB_20151127_120000_3_SEGMENT_FINAL.csv

Filename expansion in Zsh using the (n) globbing qualifier:

% for f in *(n); do echo "$f"; done
AB_20151127_120000_0_SEGMENT_FINAL.csv
AB_20151127_120000_1_SEGMENT_FINAL.csv
AB_20151127_120000_2_SEGMENT_FINAL.csv
AB_20151127_120000_3_SEGMENT_FINAL.csv
AB_20151127_120000_10_SEGMENT_FINAL.csv
kos
  • 41,378