Could libguestfs use springfield?

Could libguestfs use springfield?

[Richard W.M. Jones]

libguestfs provides a C library and large set of tools for
manipulating disk images.  http://libguestfs.org/

As part of this we provide APIs to open VM disks and do things like
enumerate partitions or resize logical volumes.  The actual way this
works currently is we run the external commands (eg. parted, lvresize)
inside a small virtual machine and pass the right command line options
or parse the output.  In some cases we're also parsing stuff out of
kernel /sys/block.

We've accumulated a large amount of code to do this (I counted 60619
lines of code in the current version).  Here are a few examples so you
can see in concrete terms what I'm talking about:

https://github.com/libguestfs/libguestfs/blob/afd1c70601c51043684a0245ce2f63d71a9cc07a/daemon/parted.c#L344

https://github.com/libguestfs/libguestfs/blob/afd1c70601c51043684a0245ce2f63d71a9cc07a/daemon/lvm.c#L271

Steven W pointed me to "Project Springfield" and it sort of looks like
it's in the same area.  Could libguestfs replace the parsing code
above with this?

What might be problems: We have no python or dbus in the appliance.
So anything that depends on those is a non-starter.

TBH the project webpage left me more confused than enlightened.  There
seem to be lots of projects (subprojects?) doing stuff with odd names
and no unifying philosophy, and I'm not sure if Project Springfield is
a thing or more of an intention.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v

Re: Could libguestfs use springfield?

[Vojtěch Trefný]

On 06/14/2018 01:15 PM, Richard W.M. Jones wrote:
> 
> libguestfs provides a C library and large set of tools for
> manipulating disk images.  http://libguestfs.org/
> 
> As part of this we provide APIs to open VM disks and do things like
> enumerate partitions or resize logical volumes.  The actual way this
> works currently is we run the external commands (eg. parted, lvresize)
> inside a small virtual machine and pass the right command line options
> or parse the output.  In some cases we're also parsing stuff out of
> kernel /sys/block.
> 
> We've accumulated a large amount of code to do this (I counted 60619
> lines of code in the current version).  Here are a few examples so you
> can see in concrete terms what I'm talking about:
> 
> https://github.com/libguestfs/libguestfs/blob/afd1c70601c51043684a0245ce2f63d71a9cc07a/daemon/parted.c#L344
> 
> https://github.com/libguestfs/libguestfs/blob/afd1c70601c51043684a0245ce2f63d71a9cc07a/daemon/lvm.c#L271
> 
> Steven W pointed me to "Project Springfield" and it sort of looks like
> it's in the same area.  Could libguestfs replace the parsing code
> above with this?
> 
> What might be problems: We have no python or dbus in the appliance.
> So anything that depends on those is a non-starter.
> 
> TBH the project webpage left me more confused than enlightened.  There
> seem to be lots of projects (subprojects?) doing stuff with odd names
> and no unifying philosophy, and I'm not sure if Project Springfield is
> a thing or more of an intention.
> 
> Rich.
> 

Hi, project springfield is a collection of existing projects, 
libblockdev is probably the "subproject" you are looking for. It is a 
plugin based C library -- we have plugins for working with btrfs, 
filesystems (ext, xfs, vfat and ntfs), lvm, mdraid, partitions etc.
It mostly also uses the command line tools and in some cases also other 
existing libraries (libcryptsetup, libmount, libparted...).

Libblockdev Github repo: https://github.com/storaged-project/libblockdev
and API documentation: http://storaged.org/libblockdev/

 From the code you sent it looks like libblockdev covers most of the 
functionality libguestfs needs. Some functionality is missing (e.g. we 
don't support changing of uuid for lvs) but we can add these missing bits.

--
Vojtech Trefny

Re: Could libguestfs use springfield?

[Richard W.M. Jones]

[Adding libguestfs mailing list]

On Thu, Jun 14, 2018 at 04:15:57PM +0200, Vojtěch Trefný wrote:
> 
> 
> On 06/14/2018 01:15 PM, Richard W.M. Jones wrote:
> >
> >libguestfs provides a C library and large set of tools for
> >manipulating disk images.  http://libguestfs.org/
> >
> >As part of this we provide APIs to open VM disks and do things like
> >enumerate partitions or resize logical volumes.  The actual way this
> >works currently is we run the external commands (eg. parted, lvresize)
> >inside a small virtual machine and pass the right command line options
> >or parse the output.  In some cases we're also parsing stuff out of
> >kernel /sys/block.
> >
> >We've accumulated a large amount of code to do this (I counted 60619
> >lines of code in the current version).  Here are a few examples so you
> >can see in concrete terms what I'm talking about:
> >
> >https://github.com/libguestfs/libguestfs/blob/afd1c70601c51043684a0245ce2f63d71a9cc07a/daemon/parted.c#L344
> >
> >https://github.com/libguestfs/libguestfs/blob/afd1c70601c51043684a0245ce2f63d71a9cc07a/daemon/lvm.c#L271
> >
> >Steven W pointed me to "Project Springfield" and it sort of looks like
> >it's in the same area.  Could libguestfs replace the parsing code
> >above with this?
> >
> >What might be problems: We have no python or dbus in the appliance.
> >So anything that depends on those is a non-starter.
> >
> >TBH the project webpage left me more confused than enlightened.  There
> >seem to be lots of projects (subprojects?) doing stuff with odd names
> >and no unifying philosophy, and I'm not sure if Project Springfield is
> >a thing or more of an intention.
> >
> >Rich.
> >
> 
> Hi, project springfield is a collection of existing projects,
> libblockdev is probably the "subproject" you are looking for. It is
> a plugin based C library -- we have plugins for working with btrfs,
> filesystems (ext, xfs, vfat and ntfs), lvm, mdraid, partitions etc.
> It mostly also uses the command line tools and in some cases also
> other existing libraries (libcryptsetup, libmount, libparted...).
> 
> Libblockdev Github repo: https://github.com/storaged-project/libblockdev
> and API documentation: http://storaged.org/libblockdev/
> 
> From the code you sent it looks like libblockdev covers most of the
> functionality libguestfs needs. Some functionality is missing (e.g.
> we don't support changing of uuid for lvs) but we can add these
> missing bits.
> 
> --
> Vojtech Trefny

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html

Re: Could libguestfs use springfield?

[Steven Whitehouse]

Hi,


On 14/06/18 15:43, Richard W.M. Jones wrote:
> [Adding libguestfs mailing list]
>
> On Thu, Jun 14, 2018 at 04:15:57PM +0200, Vojtěch Trefný wrote:
>>
>> On 06/14/2018 01:15 PM, Richard W.M. Jones wrote:
>>> libguestfs provides a C library and large set of tools for
>>> manipulating disk images.  http://libguestfs.org/
>>>
>>> As part of this we provide APIs to open VM disks and do things like
>>> enumerate partitions or resize logical volumes.  The actual way this
>>> works currently is we run the external commands (eg. parted, lvresize)
>>> inside a small virtual machine and pass the right command line options
>>> or parse the output.  In some cases we're also parsing stuff out of
>>> kernel /sys/block.
>>>
>>> We've accumulated a large amount of code to do this (I counted 60619
>>> lines of code in the current version).  Here are a few examples so you
>>> can see in concrete terms what I'm talking about:
>>>
>>> https://github.com/libguestfs/libguestfs/blob/afd1c70601c51043684a0245ce2f63d71a9cc07a/daemon/parted.c#L344
>>>
>>> https://github.com/libguestfs/libguestfs/blob/afd1c70601c51043684a0245ce2f63d71a9cc07a/daemon/lvm.c#L271
>>>
>>> Steven W pointed me to "Project Springfield" and it sort of looks like
>>> it's in the same area.  Could libguestfs replace the parsing code
>>> above with this?
>>>
>>> What might be problems: We have no python or dbus in the appliance.
>>> So anything that depends on those is a non-starter.
>>>
>>> TBH the project webpage left me more confused than enlightened.  There
>>> seem to be lots of projects (subprojects?) doing stuff with odd names
>>> and no unifying philosophy, and I'm not sure if Project Springfield is
>>> a thing or more of an intention.
>>>
>>> Rich.
>>>
>> Hi, project springfield is a collection of existing projects,
>> libblockdev is probably the "subproject" you are looking for. It is
>> a plugin based C library -- we have plugins for working with btrfs,
>> filesystems (ext, xfs, vfat and ntfs), lvm, mdraid, partitions etc.
>> It mostly also uses the command line tools and in some cases also
>> other existing libraries (libcryptsetup, libmount, libparted...).
>>
>> Libblockdev Github repo: https://github.com/storaged-project/libblockdev
>> and API documentation: http://storaged.org/libblockdev/
>>
>>  From the code you sent it looks like libblockdev covers most of the
>> functionality libguestfs needs. Some functionality is missing (e.g.
>> we don't support changing of uuid for lvs) but we can add these
>> missing bits.
>>
>> --
>> Vojtech Trefny

If there is scope for cooperation in this area, then that is very much 
what we hope to achieve with Springfield. I'm sure that we'll find that 
there are a number of projects that we were not initially aware of that 
might be good areas for collaboration/inspiration as we progress. So 
that is one reason for the slightly vague description at the moment - I 
expect this to tighten up a bit as things mature.

That said we do want to make sure that the docs on the web site are 
helpful in explaining what Springfield is all about, so suggestions are 
very welcome as to how it could be improved. Are there specific parts 
that are confusing?

One reason that there has not yet been the overall unification of 
subprojects is that this will take time. Initially there are quite a 
large number of "obvious" gaps in the current tooling, so a lot of the 
initial work has just been in addressing those gaps. One thing that Paul 
Evans has been working on, for example, is to ensure that there is good 
filesystem support for PCP, since that is a good way to get a 
performance monitoring framework up and running quickly.

Jan Tulak started looking at how we might look to provide a unified 
interface to the various filesystem utilities, however that is a tricky 
area, since it requires buy-in from the various maintainers of the 
utilities and there is a bit of chicken-and-egg in relation to wanting 
something to be useful before adopting it. I'm sure we'll get there in 
the end, but it will take some time to agree the right approach.

Certainly libguestfs sounds like the kind of project that might be 
useful to integrate with Springfield. One way to help figure that out 
would be to ask some questions....

  - Are there features which libguestfs requires, which have perhaps 
already been solved in one of the Springfield subprojects?
  - Are there features which Springfield should have, but which are 
already solved in libguestfs?

So sharing todo lists/requirements would be a good start in looking for 
common ground.

On the technology side, I don't think that there are any hard and fast 
rules on what we use in Springfield. My own hope is that it will remain 
modular enough that we can easily change components if required - as 
with any project, not every decision made at the start will necessarily 
be optimal, so my feeling is that we should design with that in mind and 
make it easy to replace/upgrade components in the future, so far as 
possible.

I hope that gives a bit more background,

Steve.