HDFS is a psudeo block interface. If you have a real filesystem like lustre, or GPFS, not only do you have the abilty to use other tools, you can use that storage for other things.
In the case of GPFS, you have configurable redundancy. Sadly with lustre, you need decent hardware, otherwise you're going to loose data.
In all these things, paying bottom dollar for hardware, forgoing support is a false economy. At scales of 1pb+ (which is about 1/2 a rack now) its much much cheaper to use off the shelf parts with 24/7 support than "softwareing" your way out.
back to the topic, HDFS is really somewhat waste of disk space, especially when used for something like munching logs
> At scales of 1pb+ (which is about 1/2 a rack now) its much much cheaper to use off the shelf parts with 24/7 support than "softwareing" your way out.
depends, if you need monthly reports from logs, as long as you don't loose storage completely, then using even second hand hardware or decommissioned from prod is cheapest choice
HDFS is a psudeo block interface. If you have a real filesystem like lustre, or GPFS, not only do you have the abilty to use other tools, you can use that storage for other things.
In the case of GPFS, you have configurable redundancy. Sadly with lustre, you need decent hardware, otherwise you're going to loose data.
In all these things, paying bottom dollar for hardware, forgoing support is a false economy. At scales of 1pb+ (which is about 1/2 a rack now) its much much cheaper to use off the shelf parts with 24/7 support than "softwareing" your way out.