Как я могу объединить несколько файлов в один без промежуточного файла? [closed]

Closed. This question is opinion-based. It is not currently accepting answers.

Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.

Closed last year.

Here is the problem I'm facing:

I am string processing a text file ~100G in size.
I'm trying to improve the runtime by splitting the file into many hundreds of smaller files and processing them in parallel.
In the end I cat the resulting files back together in order.

The file read/write time itself takes hours, so I would like to find a way to improve the following:

cat file1 file2 file3 ... fileN >> newBigFile

This requires double the diskspace as file1 ... fileN takes up 100G, and then newBigFile takes another 100Gb, and then file1... fileN gets removed.
The data is already in file1 ... fileN, doing the cat >> incurs read and write time when all I really need is for the hundreds of files to reappear as 1 file...

unix concatenation cat

задан Fabby 2 August 2019 в 13:47

0 ответов

Другие вопросы по тегам: