vim: multiple "sessions" in the same instance?

I work on a growing project with vim. I like to have most of the project files open in the same session. However, as the project's growing, I start to have too many files open at the same time; switching between files [*] starts getting a bit unproductive at 40+ files because the list gets too long (for, say, 20 files this way of navigating works very well for me, however).

Is there a way to split the session in multiple "sub-sessions" in the same instance of vim?

How I imagine it is that each session would contain, say, 20 files and only list those with :ls, and that I could switch between them in the same vim window (basicallly the same as if I'd run multiple instances of vim in the same shell and switch between them with Ctrl-Z; fg %X, just without leaving vim).

[*] I like to navigate between files with :ls followed by :bX (or directly :bX in case I've memorized a buffer number), along with :bp/:bn, and I always have at least two split open, and this workflow works quite well for me, so I'm not looking for more efficient alternatives to :ls.

Solution

I never use :ls. However I often work on big projects, and I open (many) files from different projects (and I'm still able to compile what I want in the mode I want (debug, release, ...), generate tags independently for each project, have different coding styles, etc)

Your question seems to be about "How do I go from one file to another". I have many split windows opened simultaneously and I jump from one to the other with:

:sb, which supports tab completion on filenames (already opened in buffer)
:sp **/filename*<tab> (When it's not already opened)
an old plugin of mine -- others usually use Unite, command-T or other similar plugins -- to merge :sp and :sb into one command.
Tags. The default integration of tags (/csope) is already nice. Yet, I've developed lh-tags, in order to simply the tags navigation in a world of overloaded and overridden functions as it's the case in C++.

EDIT: I've just pushed a highly experimental :Project <name> :ls feature in lh-vim-lib. Note: this new project feature wasn't meant to do what you're looking for, but it's easy to have a restricted :ls thanks to it.

To use it, you'll have to first register buffers to one project or another. Here, I recommend plugins like local_vimrc if each project can be distinguished as files under a given directory. For each root directory place a _vimrc_local.vim file in it that'll contain:

:let s:k_version = 1
" Global definitions executed everytime we enter a file belonging to the project
" This is where we set g:lobal_variables and options for project-unaware plugins.
....

" Then the anti-reinclusion guards for buffer definitions
if &cp || (exists("b:loaded__my_foobar_project_settings")
      \ && (b:loaded__my_foobar_project_settings > s:k_version)
      \ && !exists('g:force_reload__my_foobar_project_settings'))
  finish
endif
let b:loaded__my_foobar_project_settings = s:k_version
let s:cpo_save=&cpo
set cpo&vim

" HERE, we say the current buffer belongs to a project
" solution 1 when we need more control
:call lh#project#define(s:, {'name': 'YouProjectName'})
" OR solution 2 which is easier to manipulate (tab completion, ...)
:Project --define YourProjectName

You can also have a smaller granularity if you wish (this is something which is still poorly documented).

Then, you'll be able to consult the list of projects with :Project --list, or to consult the buffers associated to a given project with :Project YouProjectName :ls.