gian/seaweedfs
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(volume): cap leveldb OpenFilesCacheCapacity per index DB (#9139) (#9223)

Browse Source 
* fix(volume): cap leveldb OpenFilesCacheCapacity per index DB (#9139)

The leveldb opt.Options for NeedleMapLevelDb / Medium / Large never
set OpenFilesCacheCapacity, so each leveldb instance defaulted to
goleveldb's 500. On servers with thousands of volumes, that ceiling
stacks across DBs and exhausts even high ulimits, starving WAL
rotation:

  failed to write leveldb: open .../000006.log: too many open files

CompactionTableSizeMultiplier=10 already keeps the SST count low,
so a small per-DB cache is sufficient. Cap at 16 / 32 / 64 for the
small / medium / large variants so per-DB FD usage is bounded.

* storage: hoist leveldb FD-cap values into named constants

Per review feedback: replace the inline 16/32/64 literals with
LevelDb{,Medium,Large}OpenFilesCacheCapacity, and move the rationale
(why 500 is too high per-DB on busy servers, what the tradeoff is)
into a package-level comment so future readers see the memory vs.
performance picture at the constant declaration instead of inline.
This commit is contained in:
Chris Lu 2026-04-26 02:15:15 -07:00 committed by GitHub
parent 525900dfe4
commit 654292b57d
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 29 additions and 9 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

38
weed/storage/volume_loading.go
View File

@ -16,6 +16,23 @@ import (
"github.com/seaweedfs/seaweedfs/weed/util"
)
// Per-DB caps on goleveldb's open SST file cache. The library default is 500
// per DB, but a volume server hosts one DB per volume — easily thousands —
// so the per-DB default sums into FD exhaustion (`open .../00000N.log: too
// many open files`) even with generous ulimits, especially when leveldb is
// rotating its WAL.
//
// The trade-off: a larger cache lowers re-open overhead on cold reads, a
// smaller cache bounds total FD usage. CompactionTableSizeMultiplier=10
// already keeps SST counts low (~10x larger SSTs => ~10x fewer files), so
// even the small-volume cap is enough to keep the working set hot while
// leaving headroom for thousands of co-resident DBs.
const (
LevelDbOpenFilesCacheCapacity = 16
LevelDbMediumOpenFilesCacheCapacity = 32
LevelDbLargeOpenFilesCacheCapacity = 64
)
func loadVolumeWithoutIndex(dirname string, collection string, id needle.VolumeId, needleMapKind NeedleMapKind, ver needle.Version) (v *Volume, err error) {
v = &Volume{dir: dirname, Collection: collection, Id: id}
v.SuperBlock = super_block.SuperBlock{}
@ -196,9 +213,10 @@ func (v *Volume) load(alsoLoadIndex bool, createDatIfMissing bool, needleMapKind
}
case NeedleMapLevelDb:
opts := &opt.Options{
BlockCacheCapacity: 2 * 1024 * 1024, // default value is 8MiB
WriteBuffer: 1 * 1024 * 1024, // default value is 4MiB
CompactionTableSizeMultiplier: 10, // default value is 1
BlockCacheCapacity: 2 * 1024 * 1024, // default value is 8MiB
WriteBuffer: 1 * 1024 * 1024, // default value is 4MiB
CompactionTableSizeMultiplier: 10, // default value is 1
OpenFilesCacheCapacity: LevelDbOpenFilesCacheCapacity, // see package-level docs
}
if v.tmpNm != nil {
glog.V(0).Infoln("updating leveldb index", v.FileName(".ldb"))
@ -211,9 +229,10 @@ func (v *Volume) load(alsoLoadIndex bool, createDatIfMissing bool, needleMapKind
}
case NeedleMapLevelDbMedium:
opts := &opt.Options{
BlockCacheCapacity: 4 * 1024 * 1024, // default value is 8MiB
WriteBuffer: 2 * 1024 * 1024, // default value is 4MiB
CompactionTableSizeMultiplier: 10, // default value is 1
BlockCacheCapacity: 4 * 1024 * 1024, // default value is 8MiB
WriteBuffer: 2 * 1024 * 1024, // default value is 4MiB
CompactionTableSizeMultiplier: 10, // default value is 1
OpenFilesCacheCapacity: LevelDbMediumOpenFilesCacheCapacity, // see package-level docs
}
if v.tmpNm != nil {
glog.V(0).Infoln("updating leveldb medium index", v.FileName(".ldb"))
@ -226,9 +245,10 @@ func (v *Volume) load(alsoLoadIndex bool, createDatIfMissing bool, needleMapKind
}
case NeedleMapLevelDbLarge:
opts := &opt.Options{
BlockCacheCapacity: 8 * 1024 * 1024, // default value is 8MiB
WriteBuffer: 4 * 1024 * 1024, // default value is 4MiB
CompactionTableSizeMultiplier: 10, // default value is 1
BlockCacheCapacity: 8 * 1024 * 1024, // default value is 8MiB
WriteBuffer: 4 * 1024 * 1024, // default value is 4MiB
CompactionTableSizeMultiplier: 10, // default value is 1
OpenFilesCacheCapacity: LevelDbLargeOpenFilesCacheCapacity, // see package-level docs
}
if v.tmpNm != nil {
glog.V(0).Infoln("updating leveldb large index", v.FileName(".ldb"))