Andrew Pollack's Blog

Technology, Family, Entertainment, Politics, and Random Noise

Cutting through the B.S. -- Should you deploy DAOS right now or not?

By Andrew Pollack on 02/24/2009 at 10:18 PM EST

As posted on Ed's blog, there's a lot of talk going on right now about DAOS. I thought I'd try to clarify some of the benefits and potential pitfalls as clearly as possible to help some of you make the decision.

First the Potential Benefits

Most sites are reporting between 40 and 60 percent reduction in disk storage requirements without making any policy changes or requiring users to change behavior. I personally believe you will be hard pressed to find any site with more than 50 mail users that isn't going to see similar savings unless that site already has some pretty stringent rules in place or unusual work patterns.

Saving 50% of disk space can mean a proportional savings in disk I/O -- the number one determining factor in server performance.

Design partners have been heavily testing DAOS for quite a while, and people I know and respect a great deal have really tortured it by crashing servers and even purposely damaging files to see how robust it is. It's proving to be very reliable under real world conditions.

DAOS is the future. IBM could have done it behind the scenes and hidden it from you -- just called it the newest ODS (On Disk Structure) version. That's not the way IBM does things, but don't mistake the fact that DAOS is going to be key to many additional space and performance saving features in the future. As a Design Partner, I've been privy to some of those things and am only allowed to say "wow".

Now the Potential Pitfalls

Gone for good are the days when you could just copy an NSF file at the operating system from one machine to another. You shouldn't be doing that now, but you surely can't do it with databases using DAOS

Backup and Restore is more complex. There is no getting around this point. For now, best practices and tested solutions haven't really emerged around this issue. There are tools which support it -- Tivoli and Symantec both have backup products that fully support Domino 8.5. I'm just not sure enough time has passed that we know all the issues. For mission critical databases -- at least for now -- I'm replicating to a non DAOS server and taking backups from that.

Those are the issues as I see them. A few people are out there claiming that "well run" environments won't see savings because the "pruning" process is batched rather than done in real time. I completely disagree. I think its a fallacious argument based on totally incorrect assumptions. We'll see if someone can prove me wrong -- but I'm not holding my breath.



There are  - loading -  comments....

Great post AndrewBy Bruce Elgort on 02/24/2009 at 11:22 PM EST
Thanks for posting this. Great info on DAOS.
re: Cutting through the B.S. -- Should you deploy DAOS right now or not?By Chuck Hauble on 02/25/2009 at 07:50 AM EST
Hey Andrew,

Symantec is telling me that their NetBackup Datacenter product does not support
8.5... are you talking about a different product?
re: Cutting through the B.S. -- Should you deploy DAOS right now or not?By Jim Casale on 02/25/2009 at 08:26 AM EST
If their product supports API level backups for 8. it should work for 8.5 too
re: Cutting through the B.S. -- Should you deploy DAOS right now or not?By Jim Casale on 02/25/2009 at 08:25 AM EST
Andrew, When Tivoli and Symantec say their backups are certified for 8.5 I
think they are just talking about regular NSF backups, not specifically backing
up the NLO files in one fell swoop. For the benefits you get it shouldn't be a
deal breaker to add file level backup to your normal backup routine. Backups
can be scheduled so the only time you have to do anything extra is when you are
doing a restore.

For what it's worth, if you are backing up the replicated NSF you are not
getting the full benefits of DAOS but never hurts to err on the safe side.
re: Cutting through the B.S. -- Should you deploy DAOS right now or not?By Paul Gagnon on 02/25/2009 at 10:20 AM EST
"if you are backing up the replicated NSF you are not getting the full benefits
of DAOS"

Jim, that is certainly true if all you have is a single mail server or they are
all on the same lan segment. However, for my 6 mail servers all
geographically separated, DAOS makes sense at each facility and will save me a
bundle in space. Instead of running 6 different backups, all my mail is
replicated to a single server at our DR site, each group of mail files under
their own folder under mail using the server name.

mail\server1
mail\server2
mail\server3
etc

The replication is scheduled and runs after regular business hours, then the
Galaxy does both full nsf and document level backups on that domino server and
sends it to tier2 de-dupe eqipment, then after a time can be offloaded to tier3
media such as tape or optical for longer retentions.

I currently have no reason to run DAOS on the backup Domino server, but when
support for DAOS is added to my backup software, then I can see that DAOS would
have even greater space savings on that box than all of my other mail servers,
because of all the email attachments that get sent between sites. It's a
win-win for me in the long run.

Good writeup Andrew, thanks.
well saidBy Andrew Pollack on 02/25/2009 at 10:21 AM EST
You've done a good job on this reply laying out points I should have made as
well.
I agreeBy Andrew Pollack on 02/25/2009 at 10:20 AM EST
I agree that replicating to a full NSF and backing up from there is not getting
the best advantage of DAOS. As an early adopter and part of the design
partner program, I'm frequently running internal versions of both server and
client software that are "Code Drops" -- not even real beta code. Most is
pretty good, some isn't. As a result, I try to take a very conservative
approach.
re: Cutting through the B.S. -- Should you deploy DAOS right now or not?By Dan King on 02/25/2009 at 08:52 AM EST
So, what you're saying is that you've saved a load of space on your main server
but have to set up a new one purely to do the backups. Not sure how this is
saving space....

I was looking forward to DAOS but hadn't looked into it properly. Your post
makes it less interesting to be honest. Hard drive space is cheap, backup space
and strategy was what I thought would push DAOS through.

So maybe you can clear it up (or someone can). We currently just use arcserve
to do file backup and can restore any mail file from our nightly full backups
(we're a fairly small company). How will DAOS work in this scenario?

I was figuring our nightly backups would decrease in size potentially saving
the next upgrade of our backup strategy that happens every x years. With a
restore using DAOS we'd have to restore the DAOS store as well as a mail file I
guess? And this is only if the relevant attachment had been removed from the
DAOS store, that could be tested first? Not ideal and slower to restore single
mail files, but with the benefits of space saving still present.
That's why I posted what I did, Dan.By Andrew Pollack on 02/25/2009 at 10:24 AM EST
I posted exactly the way I did for just this reason.

If you fully buy into DAOS, you should be perfectly able to back up your DAOS
drives as you describe and gain the benefits you expect. My caution is that
until we've got some time and experience applied, extra testing is important --
you have to assess the value and risk as well as your own time to test your
backup and restore processes.

The benefits are there -- and the risks SHOULD be minimal. I have no doubt
that within a short time we'll have established best methods and practices for
them. For now, this seems to me the one area of risk.
re: That's why I posted what I did, Dan.By Dan King on 02/25/2009 at 10:57 AM EST
OK, thanks for the reply Andrew, just wanted to confirm my thoughts
re: Cutting through the B.S. -- Should you deploy DAOS right now or not?By Max on 02/25/2009 at 10:14 AM EST
The backup and restore process for DAOS enabled .nsf files is very well
described in the Notes and Domino wiki:

http://www-10.lotus.com/ldd/dominowiki.nsf/dx/daos-backup-and-restore

But that's only the how-to. The 'real' consideration in my oppinon is the
capability of your backup infrastructure. Backup and restore of .nsf's usually
means a large amount of data in a few very large files. And modern tape drives
are really good at that. You can easily write/read 200GB per hour of .nsf
files directly to/from tape.

Now, moving to DAOS means smaller .nsf's but a brazillion of very small files
in your file system. (In a TB of .nsf files we had almost 6 million
attachments.) Once detached as .nlo this is static data, but you still have to
incrementally backup a filesystem with millions of files. And you might run
into problems here. Tape drives aren't doing well in storing many small tiny
files. You might not have a VTL or disk based backup. And in case of a desaster
recovery you will have to restore those millions of files all at once.

And here's the catch. Depending on your backup hard/software restoring 300GB of
millions of .nlo's will very likely take much (and I mean MUCH) longer than
restoring a TB of several hundreds of .nsf files.

I'm not saying DAOS in terms of data backup is a bad thing. Quite contrary,
it's quite tempting actually. But like always you have to test that very
carefully. And not only the backups - test the restores!
Yes, these are all key issues.By Andrew Pollack on 02/25/2009 at 10:26 AM EST
Exactly the reason why standards and best practices need to develop. One
solution may be to tar the backups to a single large file first, then archive
that file to tape. The risk to that of course is that a single large file on
tape is more easily damaged. There are a thousand variations to consider.
re: Yes, these are all key issues.By Max on 02/25/2009 at 10:47 AM EST
Jepp. The footprint of your backup will change to less data with considerably
more objects. If your infrastructure isn't up to that you'll have to invest in
something new.

Same applies to your filesystem. On UNIX boxes the fs is probably optimised for
few, large files. Most likely you'll need a new vg for the nlo's. Those disks
have to come from somewhere.

Once again, I think DAOS is the way to go. And it was about time. But as robust
it is, we'll need experience in lage enterprise environments.
re: Yes, these are all key issues.By Jim Casale on 02/25/2009 at 12:26 PM EST
One of the advantages of DAOS as I see it is that you will not have to backup
the same data over and over again. If you do weekly full backups of Domino data
then we can assume about 40-60% of that is attachments.Using DAOS means you
will not be backing up those attachments every week. Since the NLO file doesn't
change once it is stripped out of the email then why back it up every week? I
can see monthly full backups but not weekly backups of the NLO files.

I do see issues with restores for individual database where you have to restore
thousands of files. Assuming you do monthly full backups and incrementals in
between you still need to have those files available when you try and restore
them. That means you need to have enough capacity in your backup system to keep
those files onsite. If you are a large TSM shop this is a non-issue but for
smaller shops it means recalling a bunch of tapes for the restore.
re: Yes, these are all key issues.By Max on 02/25/2009 at 03:05 PM EST
Jim, you are absolutely right. The nlo's don't change. If you use TSM they are
actually only backuped once, and never again. Because TSM uses something called
progressive backup - which translates into 'incremental forever'. So, the
amount of data in the daily backup will decrease dramatically, even more than
the savings in diskspace.

But on a large server you will end up with millions of nlo's. And the (even)
incremental backup of a filesystem with millions of files isn't trivial. You
(the backup software) have to figure out which few of those files are new and
need to be copied to tape. In TSM you can use something called journaled
backup, with a background process logging the new and changed files eligible
for backup. But I've no idea how well that plays with Domino. That needs to be
tested.

And, the backup performance im terms of GB/h for small files is much less then
for large files. That's why on fileservers you have tools like volume based or
image backup. Making a block based image of the fs rather than dealing with the
single files.

The thing is, the characteristics of your server will change from a database
server more to a file server. And I think that many Domino admins will have
little or no experience with large filesystems with many files. E.g. you will
have to reconsider running large servers on Windows, because if yor little box
blue screens and boots into a disk error checking you are doomed. A chkdsk on a
NTFS filesystem with millions of files might run for several days. It's not so
much the volume but the number of objects that counts here. Ask your file
server guys - they buy NetApps and EMCs for a reason.

Andrew really has a point with this post. Enabeling DAOS is not just: 'Wow! I
can save a 400GB of diskspace!' There comes a lot more with it. Nevertheless,
undoubted DAOS is the future, and we have to get expirienced with it. The
discussion is highly appreciated.

BTW: With TSM the restore of the nlo's for a single database is a breathe.
Because domino can build a list of the missing nlo's and TSM can use this list
directly for a restore - see the wiki for details.
re: Yes, these are all key issues.By Jim Casale on 02/25/2009 at 04:33 PM EST
You make some good points - especially on the numbers of files to be backed up.
I don't think it is a deal breaker though as I had backed up Commonstore files
on a W2K server with no issues. The data drive had over 1 million files. I
would do full backups monthly and incremental daily.

Also, the TSM restore is only a breeze if the file data is available onsite and
in one of TSM's storage pools. If not you need to recall tapes,etc.
re: Yes, these are all key issues.By Max on 02/26/2009 at 06:13 AM EST
The .nlo files should be available in a primary storage pool onsite. If you
need to recall offsite tapes for a restore, go and shoot your TSM admin...

The backup of the .nlo's with TSM will require some consideration, probably a
new management class and an additional shedule - nothing that causes your TSM
admin a headache. If you use TSM and TDP for Domino with archived transaction
logs your TSM configuration is already complex anyway. You will already have
different management classes, storage pools and shedules for the different
objects. The backup of the .nlo's adds up to that - but not in a considerable
way.

I don't think that the number of .nlo's will affect the filesystem performance
on most servers. The interesting question is: where's the threshold here?

3 million, 4 million, 5 million.... when will your server start to suffer? Can
you predict that? To my knowlege there aren't any recommendations out on the
max. numbers.
re: Yes, these are all key issues.By Jim Casale on 02/26/2009 at 09:26 AM EST
"The .nlo files should be available in a primary storage pool onsite. If you
need to recall offsite tapes for a restore, go and shoot your TSM admin..."

That one had me rolling on the floor...oh crap..wait a minute..I was the TSM
admin...

Long story short I wasn't able to keep data in the primary storage pool because
I could never convince them to fund the system properly..but I always had copy
tapes onsite
re: Yes, these are all key issues.By Max on 02/26/2009 at 09:39 AM EST
shoot your employer....
re: Cutting through the B.S. -- Should you deploy DAOS right now or not?By Chuck Hauble on 02/25/2009 at 10:15 AM EST
Works, supported/certified are very different things in my mind.

Jim I think you are right about it working with the base nsf files without
issue. In fact our testing shows that. I'm waiting to hear details back from
our Symantec tech rep on how 8.5 will be officially supported.
re: Cutting through the B.S. -- Should you deploy DAOS right now or not?By Nathan T. Freeman on 02/25/2009 at 05:30 PM EST
"A few people are out there claiming that "well run" environments won't see
savings because the "pruning" process is batched rather than done in real time."

REALLY!??! Who?
re: Cutting through the B.S. -- Should you deploy DAOS right now or not?By Alexander Torchy on 02/26/2009 at 09:30 AM EST
Sounds good... I have few questions:
What if I want to quicly restore a single DB to just view one email having an
attachment? Is is really as easy as before (I only had to copy via file system
the .nsf and go to the document)?

Does it consume extra CPU?

Will it save replication time?

If I send an email with attachment to another server having DAOS, will the
entire email be wired or just the email without the attachment?
re: Cutting through the B.S. -- Should you deploy DAOS right now or not?By Amando Fali■as on 02/26/2009 at 10:59 AM EST
What if I want to quicly restore a single DB to just view one email having an
attachment? Is is really as easy as before (I only had to copy via file system
the .nsf and go to the document)?
> Not as easy... you will need to restore the .nsf then launch a command to see
which .nlo belongs to this restored .nsf then you will need to get all .nlo and
restore them, and finally you will need to resynchronize DAOS. The whole
operation could take some hours (specially the recync) depending on the number
of .nlo files you have.
I recommend restoring the .nsf and then only if really needed restore the .nlo
files...

Does it consume extra CPU?
>As far as I know it is moreless 35% (20% for transaction logging - you should
already have it, plus 10-15% for DAOS encryption/decryption). However it
strongly depends on the environment.

Will it save replication time?
>No... when the replication occurs, DAOS will reembed the attachment inside the
document, the whole document will sent, and if the target server is
DAOS-enabled then the target server will take off the attachment from this
document and include it in its own repository. DAOS cannot share .nlo files
between servers mainly because .nlo files are encrypted using the server key.

If I send an email with attachment to another server having DAOS, will the
entire email be wired or just the email without the attachment?
>Same answer than before (the whole email will be wired).

So you will not get benefit from DAOS in the following situations:
Server is mainly an SMTP server... here attachments of outgoing emails will
be put in the repository while they are not in any .nsf. In this case you can
remove DAOS from the mail.box, but this will decrease the advantages of DAOS

Each user receives emails encrypted... in this case as the encryption is
performed by each user's key then each attachment will be UNIQUE... so if you
send an email to 50 internal users DAOS will generate 50 separate .nlo files

HTH
re: Cutting through the B.S. -- Should you deploy DAOS right now or not?By Joseph Muller on 02/26/2009 at 11:05 AM EST
ups... in my organization we have the following setting:
"When receiving unencrypted email, encrypt before storing in your mail file"

Does it mean that DAOS will not really work here?
re: Cutting through the B.S. -- Should you deploy DAOS right now or not?By Amando Fali■as on 02/26/2009 at 11:29 AM EST
Well, it depends on what you mean by "work". You won't obtain any saving at all
since each attachment will be unique for DAOS.
re: Cutting through the B.S. -- Should you deploy DAOS right now or not?By Max on 02/26/2009 at 11:33 AM EST
nope. notes doesn't encrypt each mail seperately with the users public key.

the message is encrypted with a random key and then the random key is stored
encrypted for each recipient in the mail.

if you sent an attachment encrypted to 50 users it's the same for all.


Other Recent Stories...

  1. 01/26/2023Better Running VirtualBox or VMWARE Virtual Machines on Windows 10+ Forgive me, Reader, for I have sinned. I has been nearly 3 years since my last blog entry. The truth is, I haven't had much to say that was worthy of more than a basic social media post -- until today. For my current work, I was assigned a new laptop. It's a real powerhouse machine with 14 processor cores and 64 gigs of ram. It should be perfect for running my development environment in a virtual machine, but it wasn't. VirtualBox was barely starting, and no matter how many features I turned off, it could ...... 
  2. 04/04/2020How many Ventilators for the price of those tanks the Pentagon didn't even want?This goes WAY beyond Trump or Obama. This is decades of poor planning and poor use of funds. Certainly it should have been addressed in the Trump, Obama, Bush, Clinton, Bush, and Reagan administrations -- all of which were well aware of the implications of a pandemic. I want a military prepared to help us, not just hurt other people. As an American I expect that with the ridiculous funding of our military might, we are prepared for damn near everything. Not just killing people and breaking things, but ...... 
  3. 01/28/2020Copyright Troll WarningThere's a copyright troll firm that has automated reverse-image searches and goes around looking for any posted images that they can make a quick copyright claim on. This is not quite a scam because it's technically legal, but it's run very much like a scam. This company works with a few "clients" that have vast repositories of copyrighted images. The trolls do a reverse web search on those images looking for hits. When they find one on a site that looks like someone they can scare, they work it like ...... 
  4. 03/26/2019Undestanding how OAUTH scopes will bring the concept of APPS to your Domino server 
  5. 02/05/2019Toro Yard Equipment - Not really a premium brand as far as I am concerned 
  6. 10/08/2018Will you be at the NYC Launch Event for HCL Domino v10 -- Find me! 
  7. 09/04/2018With two big projects on hold, I suddenly find myself very available for new short and long term projects.  
  8. 07/13/2018Who is HCL and why is it a good thing that they are now the ones behind Notes and Domino? 
  9. 03/21/2018Domino Apps on IOS is a Game Changer. Quit holding back. 
  10. 02/15/2018Andrew’s Proposed Gun Laws 
Click here for more articles.....


pen icon Comment Entry
Subject
Your Name
Homepage
*Your Email
* Your email address is required, but not displayed.
 
Your thoughts....
 
Remember Me  

Please wait while your document is saved.