Google Summer of Code 2010/QMP: Difference between revisions

From QEMU
No edit summary
No edit summary
 
(32 intermediate revisions by the same user not shown)
Line 1: Line 1:
== Summary ==
= Summary =
qemu has many commands attached to the monitor interface that are currently inaccessible over QMP (QEMU Machine Protocol). I intend to give continuity to this effort, converting the remaining functions to the QObject API. While doing this conversion, the QObject API can be updated to improve its functionality when required.
qemu has many commands attached to the monitor interface that are currently inaccessible over QMP (QEMU Machine Protocol). I intend to give continuity to this effort, converting the remaining functions to the QObject API. While doing this conversion, the QObject API can be updated to improve its functionality when required.


More details about the protocol: http://www.linux-kvm.org/page/MonitorProtocol
More details about the protocol: http://www.linux-kvm.org/page/MonitorProtocol
 
= Team =
== Team ==
== Student ==
=== Student ===
* '''Name:''' [[User:Miguel|Miguel Di Ciurcio Filho]]
* '''Name:''' [[User:Miguel|Miguel Di Ciurcio Filho]]
* '''Email:''' miguel.filho@gmail.com
* '''Email:''' miguel.filho@gmail.com


=== Mentor ===
== Mentor ==
* '''Name:''' [[User:LuizCapitulino|Luiz Capitulino]]
* '''Name:''' [[User:LuizCapitulino|Luiz Capitulino]]
* '''Email:''' lcapitulino@redhat.com
* '''Email:''' lcapitulino@redhat.com


== Subsystems being worked on ==
 
= GSoC Summary =
 
== query-netdev ==
 
The '''query-netdev''' command is used to show information about the current backend network devices assigned to a VM.
 
* Status: mentor has merged it into his tree.
* Revisions:
** v10: http://lists.gnu.org/archive/html/qemu-devel/2010-06/msg03289.html
** v9: http://lists.gnu.org/archive/html/qemu-devel/2010-06/msg02820.html
** v8: http://lists.gnu.org/archive/html/qemu-devel/2010-06/msg02629.html
** v7: http://www.mail-archive.com/qemu-devel@nongnu.org/msg35464.html
** v6 (now query-netdev): http://www.mail-archive.com/qemu-devel@nongnu.org/msg35464.html
** v5 (query-netdevices proposal): http://www.mail-archive.com/qemu-devel@nongnu.org/msg34634.html
** v4 (conversion of ''info network''): http://lists.gnu.org/archive/html/qemu-devel/2010-05/msg01567.html
** v3 (conversion of ''info network''): http://www.mail-archive.com/qemu-devel@nongnu.org/msg28815.html
** v2 (conversion of ''info network''): http://www.mail-archive.com/qemu-devel@nongnu.org/msg29202.html
** v1 (conversion of ''info network''): http://www.mail-archive.com/qemu-devel@nongnu.org/msg29768.html
 
== query-available-devices ==
The '''query-available-devices''' describes the capabilities of all devices registered with qdev.
 
* Status: mentor has merged it into his tree.
* Revisions:
** v4 (renamed to query-available-devices): http://lists.gnu.org/archive/html/qemu-devel/2010-07/msg01400.html
** v3: http://lists.gnu.org/archive/html/qemu-devel/2010-07/msg01060.html
** v2: http://lists.gnu.org/archive/html/qemu-devel/2010-07/msg00536.html
** v1 (query-qdm proposal): http://lists.gnu.org/archive/html/qemu-devel/2010-07/msg00194.html
 
== query-snapshots ==
 
* Status: some patches have been merged into upstream, but no QMP yet. These cleanups are needed before trying to specify the protocol.
* Patches merged upstream:
** savevm: Really verify if a drive supports snapshots: http://git.qemu.org/qemu.git/commit/?id=feeee5aca765606818e00f5a19d19f141f4ae365
*** v4: http://www.mail-archive.com/qemu-devel@nongnu.org/msg34707.html
*** v3: http://www.mail-archive.com/qemu-devel@nongnu.org/msg34510.html
*** v2: http://www.mail-archive.com/qemu-devel@nongnu.org/msg33867.html
*** v1: http://www.mail-archive.com/qemu-devel@nongnu.org/msg33751.html
** loadvm: improve tests before bdrv_snapshot_goto(): http://git.qemu.org/qemu.git/commit/?id=f0aa7a8b2d518c54430e4382309281b93e51981a
*** v2: http://lists.gnu.org/archive/html/qemu-devel/2010-07/msg01065.html
*** v1: http://lists.gnu.org/archive/html/qemu-devel/2010-07/msg00838.html
* Patches still missing a final review:
** Snapshots improvements v2 (merged into this all previous patches): http://lists.gnu.org/archive/html/qemu-devel/2010-08/msg00346.html
*** bdrv_snaphost_find/prevent snapshot overwriting/generate a default name: http://lists.gnu.org/archive/html/qemu-devel/2010-07/msg01701.html
*** new 'info snapshots' v2: http://lists.gnu.org/archive/html/qemu-devel/2010-07/msg01591.html
*** new 'info snapshots' v1: http://lists.gnu.org/archive/html/qemu-devel/2010-06/msg02064.html
 
* RFCs still pending:
** RFC about inheritance/uuid: http://lists.gnu.org/archive/html/qemu-devel/2010-08/msg00204.html
** QMP: new query-snapshots command: http://lists.gnu.org/archive/html/qemu-devel/2010-06/msg01083.html
 
== query-version ==
based on previous code sent to the list, it is an update to the query-version command.
* Status: pending review
* Patch: http://lists.gnu.org/archive/html/qemu-devel/2010-08/msg00473.html
 
== Other contributions ==
Bugs found:
* I tracked a problem to a commit made by Alex Williamson and reported it. He has sent a patch fixing it: http://lists.gnu.org/archive/html/qemu-devel/2010-07/msg01114.html
* Sent bug report about device_add: http://lists.gnu.org/archive/html/qemu-devel/2010-07/msg00529.html
 
Cleanup tries:
* RFC: Dumping traffic when using netdev devices: http://lists.gnu.org/archive/html/qemu-devel/2010-07/msg00914.html
* vlan cleanup: http://lists.gnu.org/archive/html/qemu-devel/2010-07/msg00675.html
 
= Subsystems being worked on =


{| cellpadding="4" cellspacing="0" border="1"
{| cellpadding="4" cellspacing="0" border="1"
! Handler
! Handler
! Status
! Status
! Who I should CC
|-
|-
| [[#do_info_netdevices|do_info_netdevices]]
| [[#do_info_netdev|do_info_netdev]]
| Sent v4 to qemu-devel on May 18, pending review.
| Done.
| Markus Armbruster <armbru@redhat.com>
|-
|-
| [[#do_savevm/do_loadvm|do_savevm/do_loadvm]]
| [[#do_savevm/do_loadvm|do_savevm/do_loadvm]]
| Planning/Studding the code.
| Sending small patches, not related to QMP yet.
| Kevin Wolf <kwolf@redhat.com>, Juan Quintela <quintela@redhat.com>, Christoph Hellwig <hch@infradead.org>
|-
| [[#query-available-devices|query-available-devices]]
| Done.
| Daniel P. Berrange <barrange@redhat.com>
|-
|-
| do_info_qdm
| [[#query-version|query-version]]
| Need to confirm libvirt's needs/priority about this.
| Pending review.
| Daniel P. Berrange <barrange@redhat.com>
|}
|}


=== do_info_netdevices ===
== do_info_netdev ==
TODO:
* Nothing, mentor has merged into his tree: http://repo.or.cz/w/qemu/qmp-unstable.git?a=search&h=HEAD&st=author&s=miguel.filho%40gmail.com :-D
 
== do_savevm/do_loadvm ==
TODO:
* [[Features/SnapshottingImprovements]]
 
== query-available-devices ==
TODO:
* Nothing, mentor has merged into his tree: http://repo.or.cz/w/qemu/qmp-unstable.git?a=search&h=HEAD&st=author&s=miguel.filho%40gmail.com :-D
 
== query-version ==
TODO:
* Pending review.
 
= Day by Day =
 
=== August 10 ===
* Updated the wiki and summarized all the work reported on the daily logs.
 
=== August 9 ===
* Sent an updated version of the 'query-version' command previously sent by Daniel Berrange: http://lists.gnu.org/archive/html/qemu-devel/2010-08/msg00473.html
 
=== August 4 ===
* Cleanup and sent v2 of the '''easy''' snapshotting improvements: http://lists.gnu.org/archive/html/qemu-devel/2010-08/msg00346.html
 
=== August 3 ===
* Participated in KVM weekly call and discussed with other devs about the inheritance/uuid issues
* Sent an RFC about inheritance/uuid: http://lists.gnu.org/archive/html/qemu-devel/2010-08/msg00204.html
* Updated daily logs
 
=== August 2 ===
* Fixed mentor and Markus suggestions, added all previous patch into the same series
 
=== July 30 ===
* Coded the snapshot inheritance and uuid support.
* Reworked the [[Features/SnapshottingImprovements]] page
 
=== July 29 ===
* Coded the snapshot inheritance and uuid support.
 
=== July 28 ===
* Finished and sent more savevm cleanups: http://lists.gnu.org/archive/html/qemu-devel/2010-07/msg01701.html
* Updated wiki pages
* Chat with mentor
 
=== July 27 ===
* Finished and sent the new version of the new 'info snapshots': http://lists.gnu.org/archive/html/qemu-devel/2010-07/msg01591.html
 
=== July 26 ===
* Coded query-version based on code from Daniel P. Berrange
* Updated the [[Features/SnapshottingImprovements]] page with much more information
* Coded a new version of my previous patch that changes the 'info snapshots' monitor command.
 
=== July 23 ===
* Coded and sent the v4 of query-available-devices (was query-qdm): http://lists.gnu.org/archive/html/qemu-devel/2010-07/msg01400.html
 
=== July 21 ===
* Created the page [[Features/SnapshottingImprovements]], to enumerate all known issues with savevm/loadvm and to try to fix them.
 
=== July 20 ===
* Spend many, many, many, hours trying to figure out random segfaults when using savevm/loadvm. I tracked the problem to a commit made by Alex Williamson and reported it. He has sent a patch fixing it: http://lists.gnu.org/archive/html/qemu-devel/2010-07/msg01114.html
 
=== July 19 ===
* Sent to qemu-devel the v3 version of query-qdm: http://lists.gnu.org/archive/html/qemu-devel/2010-07/msg01060.html
* Sent v2 of the loadvm() improvements after Kevin's review: http://lists.gnu.org/archive/html/qemu-devel/2010-07/msg01065.html
 
=== July 16 ===
* All day long pulling some hair to understand the networking code. I should have gone back to finish query-qdm.
 
=== July 15 ===
* Got the "final" feed back about the query-qdm proposal from Markus and Luiz.
* All day wrapping my mind around networking code of qemu and finally sent a draft to the list: http://lists.gnu.org/archive/html/qemu-devel/2010-07/msg00914.html It kinda works! :-D
 
=== July 14 ===
* Sent to qemu-devel the patch that improves the tests before bdrv_snapshot_goto(): http://lists.gnu.org/archive/html/qemu-devel/2010-07/msg00838.html It is small, but it took me a long time to figure out that you can't run bdrv_snapshot_goto() after saving the VM state, or things gets ugly.
* Began to code a prototype for adding a dump facility to netdevs, so we can get rid of vlans.
 
=== July 13 ===
* I've participated (actually, just listened) to the KVM call.
* Long chat with mentor.
* Began to cook a patch to do better tests before running bdrv_snapshot_goto() in load_vmstate().
 
=== July 12 ===
* Over the weekend I've began a cleanup to remove the vlan code, just for fun. It looked quite promising. So on July 12 I cooked a somewhat quick and dirty patch and sent it to qemu-devel: http://lists.gnu.org/archive/html/qemu-devel/2010-07/msg00675.html Got some great feed back. 
 
=== July 9 ===
* Finished and sent query-qdm docs v2: http://lists.gnu.org/archive/html/qemu-devel/2010-07/msg00536.html
* Sent bug report about device_add: http://lists.gnu.org/archive/html/qemu-devel/2010-07/msg00529.html
 
=== July 8 ===
* Tried to wrap my mind around qdev, not easy.
* Found some bugs when using device_add.
* Some fixes to the query-qdm spec.
* Chat with mentor.
 
=== July 7 ===
* Got some reviews from the first query-qdm patch, worked on it.
 
=== July 2 ===
* Brazil is out of the World Cup :-(
* Coded and sent the query-qdm patch: http://lists.gnu.org/archive/html/qemu-devel/2010-07/msg00194.html
* Updated the wiki.
 
=== July 1 ===
* Chat with Kevin about snapshot stuff after some difficulties and how qcow2 works.
* Chat with mentor about our goals
* Contacted Daniel P. Berrange about a previous patch he has done converting 'info qdm' and got his OK for me to continue it.
 
=== June 30 ===
* Fixed latest mentor's review on query-netdev and send it as v2 to qemu-devel: http://lists.gnu.org/archive/html/qemu-devel/2010-06/msg03289.html
* Coded/Studied the internals of QEMU snapshotting.
 
=== June 28 ===
* Got mentor's review on query-netdev, 2 dumb bugs :-( Damn!
 
=== June 23 ===
* After chatting with mentor, tap/sndbuf have been removed from the spec.
* Finished the query-netdev implementation and sent to qemu-devel: http://lists.gnu.org/archive/html/qemu-devel/2010-06/msg02820.html
 
=== June 22 ===
* Coded the query-netdev specification. I'm facing some difficulties and ended the day stuck with tap/sndbuf. It is very OS dependent, no way to check if its OK, a lot of stub functions.
* Updated the examples in the query-netdev protocol spec.
* Chat with mentor and Kevin Wolf about the patch I've sent to update the monitor output. Need to code new version.
* Log from the chat so I don't forget what I have to do:
<miguelzinho>  luiz, could you take a look at your reply? http://lists.gnu.org/archive/html/qemu-devel/2010-06/msg02398.html
<miguelzinho>  luiz, is there anything blocking/must further discuss about that patch?
<luiz___>  miguelzinho: my concern is that we're degrading the usability of the this command, as it can confuse users. But I don't  how to solve that
<miguelzinho>  luiz___, OTOH, the user is getting more information about the VM. I'm not sure that the new output is confusing.
<kwolf> miguelzinho: But the common case is that all images have the same snapshots anyway, right?
<kwolf> So maybe printing lots of information when it's really the same for each image, could be considered confusing indeed
<miguelzinho>  kwolf, how about an image that was added latter on and don't have snapshots in it?
<kwolf> miguelzinho: I don't say that it can't happen, but is it really the common case?
<miguelzinho>  kwolf, yeah, not that common
<kwolf> However, with the current implementation I think it can happen that info snapshots shows snapshots that you can't really  load. That's not optimal either.
<miguelzinho>  kwolf, that is exactly what I'm trying do improve
<kwolf> I know
<kwolf> Maybe we can merge everything into one list?
<kwolf> And add a column that says if a snapshot can be loaded (i.e. it's present on all images)?
<miguelzinho>  collect the available and valid snapshots and summarize the output?
<kwolf> Something like this, yes
<miguelzinho>  Sounds better, indeed
<kwolf> We should include the snapshots that exist only on some of the images, too, because they would be overwritten by savevm
<kwolf> And mark them clearly
<miguelzinho>  you mean snapshots that exist on all the images?
<miguelzinho>  the ones that are not available on all the images must be marked somehow, so savevm don't try do load those.
<kwolf> s/load/overwrite/ or s/savevm/loadvm/?
<kwolf> But I think we're talking about the same anyway
<miguelzinho>  both, hehehe, my bad
<miguelzinho>  kwolf, I will send a new version more towards we have agreed here.
<kwolf> Sounds good. luiz___, do you agree so far?
<luiz___>  yeah, looks good
<luiz___>  miguelzinho: what happens if the user tries to load a broken snapshot today?
<miguelzinho>  luiz___, first device has snapshots 1 2 and 3, second device has snapshots 1 and 2
<miguelzinho>  luiz___, loadvm 3
<miguelzinho>  luiz___, first device "goes" to snapshot 3, second device fails, there is no 3 there
<miguelzinho>  luiz___, I'm not sure, but the VM stays running anyway
<kwolf> miguelzinho: Don't we fail immediately since your first patch?
<miguelzinho>  kwolf, only if the device does not support snapshots at all
<kwolf> I see
<kwolf> So that's something else to improve
<miguelzinho>  luiz___, yep, VM keeps running. savevm.c:load_vmstate():1793
<luiz___>  miguelzinho: as kwolf says, would be nice to improve that too
 
=== June 21 ===
* After discussing with other devs on qemu-devel, the draft sent on June 18 was updated and resent do the list: http://lists.gnu.org/archive/html/qemu-devel/2010-06/msg02629.html
* What has changed:
** Got rid of vlans, a separate command for them will be created if needed
** Removed socket as a type (it is used to connect vlans)
** 'script', 'downscript' and 'vhostfd' are now optional when type is 'tap'
** Reworded the explanation of 'peer'
 
=== June 18 ===
* Finished and sent to qemu-devel just the updated documentation for query-netdev instead of the full patch series. So once the protocol is defined, I will send the code (although the code for the documentation is ready to ship) http://lists.gnu.org/archive/html/qemu-devel/2010-06/msg02497.html
* Chat with mentor.
 
=== June 17 ===
* Updated the wiki.
* Sent the v2 monitor patch to qemu-devel: http://lists.gnu.org/archive/html/qemu-devel/2010-06/msg02386.html
* Chat with mentor.
* Coded query-netdev fixes. Needs more testing.
 
=== June 16 ===
* Got mentor's review from query-netdev.
* Some back and forth about the last monitor patch, coded a new version.
 
=== June 15 ===
* First patch applied, yeah! http://git.qemu.org/qemu.git/commit/?id=feeee5aca765606818e00f5a19d19f141f4ae365
* Tested and sent the patch about the monitor: http://lists.gnu.org/archive/html/qemu-devel/2010-06/msg02064.html
 
=== June 11 ===
* Coded an small patch that updates the monitor output to show snapshot information of all supported devices.
 
=== June 10 ===
* Finished the new query-netdev patch. Sent to qemu-devel. FYI, this is the 5th try. http://www.mail-archive.com/qemu-devel@nongnu.org/msg35464.html
 
=== June 9 ===
* Coded and tested the new query-netdev.
* Wrestled with git rebase, now I understand it. How could I leave without it?
 
=== June 8 ===
* Sent to qemu-devel the RFC for the new QMP command query-snapshots.
 
=== June 7 ===
* A lot of back and forth over email from all sorts of people about query-netdevices and some chat with Anthony on IRC.
* Resolutions about it:
** Rename the command to query-netdev
** Show only the backend information (user, tap, socket, vde)
** Guest devices (frontend) are already available via info qtree, don't need to show them.
 
Result:
 
-> { "execute": "query-netdev" }
<- {
      "return": [
        {
            "device": "tap.0",
            "vlan": 0,
            "type": "tap",
            "info": {
              "script": "/etc/qemu-ifup",
              "downscript": "/etc/qemu-ifdown",
              "ifname": "tap0",
            },
        {
            "device": "user.0",
            "type": "user",
            "peer": "e1000.0",
            "info": {
              "net": "10.0.2.0",
              "netmask": "255.255.255.0"
            },
        },
 
=== June 4 ===
* Chat with mentor and Kevin Wolf.
* Sent to qemu-devel the RFC for the new QMP command  query-netdevices. I will wait for feedback before coding. http://www.mail-archive.com/qemu-devel@nongnu.org/msg34634.html
* Sent to mentor the RFC for the new QMP command query-snapshots before sending to qemu-devel. This might avoid some noise in the list.
* '''Lesson learned: compile qemu without --enable-debug before submitting anything, damn it!''' The do_savevm() v3 patch had an warning that was not caught by me and latter on caught by Kevin Wolf: http://www.mail-archive.com/qemu-devel@nongnu.org/msg34707.html After a few minutes chatting with him on IRC we figured that --enable-debug disables -O2, then some warnings could not be seen. Fixed and resent v4. I hope that know this think gets merged :-D
 
=== June 3 ===
* Fixed Kevin's suggestions and resent the do_savevm() patch to qemu-devel (now version 3): http://www.mail-archive.com/qemu-devel@nongnu.org/msg34510.html
 
=== May 31 thru June 2 ===
* Studied and toke some exams. Classes are over :-D


We were struggling to get the output of 'info network' exactly the same and juggling to work around the lack of a pattern when formatting VLANClientState.info_str. So we decided to add a new and clean command, and leave 'info network' as it is.
=== May 29 ===
* Fixed Markus and Kevin's suggestions and resent the do_savevm() patch to qemu-devel: http://www.mail-archive.com/qemu-devel@nongnu.org/msg33867.html Fingers crossed, this could be my first merge!


This is the latest patch:
=== May 28 ===
* Some chat with mentor.


http://lists.gnu.org/archive/html/qemu-devel/2010-05/msg01567.html
* I've submitted a patch to qemu-devel that verifies the snapshot capabilities of all devices in do_savevm() before trying to save anything at all. More details: http://www.mail-archive.com/qemu-devel@nongnu.org/msg33751.html


=== do_savevm/do_loadvm ===
=== May 27 ===
* Got mentor's review about last do_info_netdevices(). Will begin fixing things after finishing some more do_savevm/do_loadvm work.
* do_info_snapshots() now displays snapshot information of all devices.
qemu -hda ../a.qcow2 -hdb ../b.qcow2


TODO:
QEMU 0.12.50 monitor - type 'help' for more information
* Work on do_info_snapshots() to display snapshot information of all devices.
(qemu) info snapshots
* Verify snapshot capabilities of all devices in do_savevm() before trying to save anything at all.
Snapshot list from ide0-hd0:
* Update do_loadvm() do load a snapshot using an UUID.
ID  TAG      VM SIZE                DATE      VM CLOCK                                UUID                          PARENT UUID
1                1.5M 2010-05-27 12:24:45  00:00:05.640 f89787ea-c39e-4531-90cc-19d73abbc50e 00000000-0000-0000-0000-000000000000
2                1.5M 2010-05-27 12:24:52  00:00:12.258 4fb1f028-febb-464a-8e61-8abdfde9adc8 f89787ea-c39e-4531-90cc-19d73abbc50e
3                  1.5M 2010-05-27 12:25:05  00:00:24.921 faeb873c-f3fd-40ab-97de-67c6f12fd5ff 4fb1f028-febb-464a-8e61-8abdfde9adc8
Snapshot list from ide0-hd1:
ID  TAG      VM SIZE                DATE      VM CLOCK                                UUID                          PARENT UUID
1                    0 2010-05-27 11:33:11  00:00:02.781 8f5d24a0-9a92-4650-8b56-babaa7790700 00000000-0000-0000-0000-000000000000
2                    0 2010-05-27 11:53:39  00:00:12.073 9e471508-f9d9-46d6-815b-2c25b15152f3 00000000-0000-0000-0000-000000000000
3                    0 2010-05-27 11:53:40  00:00:12.867 c5a75fd0-2cea-433b-bd21-37025dff3cbd 9e471508-f9d9-46d6-815b-2c25b15152f3


==== May 26 ====
=== May 26 ===
* Got parent snapshot saving working. Using global variable, not sure if it is best option.
* Got parent snapshot saving working. Using global variable, not sure if it is best option.
* Updated bdrv_snapshot_dump() to properly show the UUID information. It looks really big.
* Updated bdrv_snapshot_dump() to properly show the UUID information. It looks really big.
Line 61: Line 393:
* Fixed extra_data_size offset handling when loading a qcow2 image with an empty extra area.
* Fixed extra_data_size offset handling when loading a qcow2 image with an empty extra area.
   
   
==== May 22 ====
=== May 22 ===
I was actually not saving anything. After closing the VM the UUID information was not written to disk. After hours pulling some hair I figured it as a typo and a wrong offset calculation when saving the QCowSnapshotHeader to disk.
I was actually not saving anything. After closing the VM the UUID information was not written to disk. After hours pulling some hair I figured it as a typo and a wrong offset calculation when saving the QCowSnapshotHeader to disk.


==== May 21 ====
=== May 21 ===
Got saving UUID information in a qcow2 file, using QCowSnapshot->extra_data_size and displaying it on 'info snapshots'. It seams we can have snapshot inheritance without breaking the file format or any old code. Now time to actually save the inheritance.
* Got saving UUID information in a qcow2 file, using QCowSnapshot->extra_data_size and displaying it on 'info snapshots'. It seams we can have snapshot inheritance without breaking the file format or any old code. Now time to actually save the inheritance.


Helpful references:
Helpful references:
* http://people.gnome.org/~markmc/qcow-image-format.html
* http://people.gnome.org/~markmc/qcow-image-format.html
* http://www.mail-archive.com/qemu-devel@nongnu.org/msg11663.html
* http://www.mail-archive.com/qemu-devel@nongnu.org/msg11663.html


==== May 19 ====
=== May 19 ===
After talking with Chris Lalancette from libvirt, we need snapshot inheritance in qemu, before converting do_savevm/do_loadvm to QMP. I've analyzed the code and I there is some hairy things to do. There are more people working on this code, so I will try to get a go before starting coding.
After talking with Chris Lalancette from libvirt, we need snapshot inheritance in qemu, before converting do_savevm/do_loadvm to QMP. I've analyzed the code and I there is some hairy things to do. There are more people working on this code, so I will try to get a go before starting coding.


== Source code ==
=== From late March thru May 18 ===
Sent patch v4 to qemu-devel.
 
We were struggling to get the output of 'info network' exactly the same and juggling to work around the lack of a pattern when formatting VLANClientState.info_str. So we decided to add a new and clean command, and leave 'info network' as it is.
 
This is the v4 patch:
 
http://lists.gnu.org/archive/html/qemu-devel/2010-05/msg01567.html


The working being done can be followed here:
The work on this area has begun late March, as a test if I could do something in GSoC. And here I am :-D There were a lot of interactions before the v4 patch.


http://github.com/miguelzinho/qemu-soc
Previous attempts:
* First try on 2010-04-03: http://www.mail-archive.com/qemu-devel@nongnu.org/msg28815.html
* Second try on 2010-04-08: http://www.mail-archive.com/qemu-devel@nongnu.org/msg29202.html
* Third try on 2010-04-15: http://www.mail-archive.com/qemu-devel@nongnu.org/msg29768.html

Latest revision as of 17:43, 11 August 2010

Summary

qemu has many commands attached to the monitor interface that are currently inaccessible over QMP (QEMU Machine Protocol). I intend to give continuity to this effort, converting the remaining functions to the QObject API. While doing this conversion, the QObject API can be updated to improve its functionality when required.

More details about the protocol: http://www.linux-kvm.org/page/MonitorProtocol

Team

Student

Mentor


GSoC Summary

query-netdev

The query-netdev command is used to show information about the current backend network devices assigned to a VM.

query-available-devices

The query-available-devices describes the capabilities of all devices registered with qdev.

query-snapshots

query-version

based on previous code sent to the list, it is an update to the query-version command.

Other contributions

Bugs found:

Cleanup tries:

Subsystems being worked on

Handler Status Who I should CC
do_info_netdev Done. Markus Armbruster <armbru@redhat.com>
do_savevm/do_loadvm Sending small patches, not related to QMP yet. Kevin Wolf <kwolf@redhat.com>, Juan Quintela <quintela@redhat.com>, Christoph Hellwig <hch@infradead.org>
query-available-devices Done. Daniel P. Berrange <barrange@redhat.com>
query-version Pending review. Daniel P. Berrange <barrange@redhat.com>

do_info_netdev

TODO:

do_savevm/do_loadvm

TODO:

query-available-devices

TODO:

query-version

TODO:

  • Pending review.

Day by Day

August 10

  • Updated the wiki and summarized all the work reported on the daily logs.

August 9

August 4

August 3

August 2

  • Fixed mentor and Markus suggestions, added all previous patch into the same series

July 30

July 29

  • Coded the snapshot inheritance and uuid support.

July 28

July 27

July 26

  • Coded query-version based on code from Daniel P. Berrange
  • Updated the Features/SnapshottingImprovements page with much more information
  • Coded a new version of my previous patch that changes the 'info snapshots' monitor command.

July 23

July 21

July 20

July 19

July 16

  • All day long pulling some hair to understand the networking code. I should have gone back to finish query-qdm.

July 15

July 14

  • Sent to qemu-devel the patch that improves the tests before bdrv_snapshot_goto(): http://lists.gnu.org/archive/html/qemu-devel/2010-07/msg00838.html It is small, but it took me a long time to figure out that you can't run bdrv_snapshot_goto() after saving the VM state, or things gets ugly.
  • Began to code a prototype for adding a dump facility to netdevs, so we can get rid of vlans.

July 13

  • I've participated (actually, just listened) to the KVM call.
  • Long chat with mentor.
  • Began to cook a patch to do better tests before running bdrv_snapshot_goto() in load_vmstate().

July 12

July 9

July 8

  • Tried to wrap my mind around qdev, not easy.
  • Found some bugs when using device_add.
  • Some fixes to the query-qdm spec.
  • Chat with mentor.

July 7

  • Got some reviews from the first query-qdm patch, worked on it.

July 2

July 1

  • Chat with Kevin about snapshot stuff after some difficulties and how qcow2 works.
  • Chat with mentor about our goals
  • Contacted Daniel P. Berrange about a previous patch he has done converting 'info qdm' and got his OK for me to continue it.

June 30

June 28

  • Got mentor's review on query-netdev, 2 dumb bugs :-( Damn!

June 23

June 22

  • Coded the query-netdev specification. I'm facing some difficulties and ended the day stuck with tap/sndbuf. It is very OS dependent, no way to check if its OK, a lot of stub functions.
  • Updated the examples in the query-netdev protocol spec.
  • Chat with mentor and Kevin Wolf about the patch I've sent to update the monitor output. Need to code new version.
  • Log from the chat so I don't forget what I have to do:
<miguelzinho>   luiz, could you take a look at your reply? http://lists.gnu.org/archive/html/qemu-devel/2010-06/msg02398.html
<miguelzinho>   luiz, is there anything blocking/must further discuss about that patch?
<luiz___>   miguelzinho: my concern is that we're degrading the usability of the this command, as it can confuse users. But I don't  how to solve that
<miguelzinho>   luiz___, OTOH, the user is getting more information about the VM. I'm not sure that the new output is confusing.
<kwolf> miguelzinho: But the common case is that all images have the same snapshots anyway, right?
<kwolf> So maybe printing lots of information when it's really the same for each image, could be considered confusing indeed
<miguelzinho>   kwolf, how about an image that was added latter on and don't have snapshots in it?
<kwolf> miguelzinho: I don't say that it can't happen, but is it really the common case?
<miguelzinho>   kwolf, yeah, not that common
<kwolf> However, with the current implementation I think it can happen that info snapshots shows snapshots that you can't really   load. That's not optimal either.
<miguelzinho>   kwolf, that is exactly what I'm trying do improve
<kwolf> I know
<kwolf> Maybe we can merge everything into one list?
<kwolf> And add a column that says if a snapshot can be loaded (i.e. it's present on all images)?
<miguelzinho>   collect the available and valid snapshots and summarize the output?
<kwolf> Something like this, yes
<miguelzinho>   Sounds better, indeed
<kwolf> We should include the snapshots that exist only on some of the images, too, because they would be overwritten by savevm
<kwolf> And mark them clearly
<miguelzinho>   you mean snapshots that exist on all the images?
<miguelzinho>   the ones that are not available on all the images must be marked somehow, so savevm don't try do load those.
<kwolf> s/load/overwrite/ or s/savevm/loadvm/?
<kwolf> But I think we're talking about the same anyway
<miguelzinho>   both, hehehe, my bad
<miguelzinho>   kwolf, I will send a new version more towards we have agreed here.
<kwolf> Sounds good. luiz___, do you agree so far?
<luiz___>   yeah, looks good
<luiz___>   miguelzinho: what happens if the user tries to load a broken snapshot today?
<miguelzinho>   luiz___, first device has snapshots 1 2 and 3, second device has snapshots 1 and 2
<miguelzinho>   luiz___, loadvm 3
<miguelzinho>   luiz___, first device "goes" to snapshot 3, second device fails, there is no 3 there
<miguelzinho>   luiz___, I'm not sure, but the VM stays running anyway
<kwolf> miguelzinho: Don't we fail immediately since your first patch?
<miguelzinho>   kwolf, only if the device does not support snapshots at all
<kwolf> I see
<kwolf> So that's something else to improve
<miguelzinho>   luiz___, yep, VM keeps running. savevm.c:load_vmstate():1793
<luiz___>   miguelzinho: as kwolf says, would be nice to improve that too

June 21

  • After discussing with other devs on qemu-devel, the draft sent on June 18 was updated and resent do the list: http://lists.gnu.org/archive/html/qemu-devel/2010-06/msg02629.html
  • What has changed:
    • Got rid of vlans, a separate command for them will be created if needed
    • Removed socket as a type (it is used to connect vlans)
    • 'script', 'downscript' and 'vhostfd' are now optional when type is 'tap'
    • Reworded the explanation of 'peer'

June 18

June 17

June 16

  • Got mentor's review from query-netdev.
  • Some back and forth about the last monitor patch, coded a new version.

June 15

June 11

  • Coded an small patch that updates the monitor output to show snapshot information of all supported devices.

June 10

June 9

  • Coded and tested the new query-netdev.
  • Wrestled with git rebase, now I understand it. How could I leave without it?

June 8

  • Sent to qemu-devel the RFC for the new QMP command query-snapshots.

June 7

  • A lot of back and forth over email from all sorts of people about query-netdevices and some chat with Anthony on IRC.
  • Resolutions about it:
    • Rename the command to query-netdev
    • Show only the backend information (user, tap, socket, vde)
    • Guest devices (frontend) are already available via info qtree, don't need to show them.

Result:

-> { "execute": "query-netdev" }
<- {
     "return": [
        {
           "device": "tap.0",
           "vlan": 0,
           "type": "tap",
           "info": {
              "script": "/etc/qemu-ifup",
              "downscript": "/etc/qemu-ifdown",
              "ifname": "tap0",
           },
        {
           "device": "user.0",
           "type": "user",
           "peer": "e1000.0",
           "info": {
              "net": "10.0.2.0",
              "netmask": "255.255.255.0"
           },
        },

June 4

  • Chat with mentor and Kevin Wolf.
  • Sent to qemu-devel the RFC for the new QMP command query-netdevices. I will wait for feedback before coding. http://www.mail-archive.com/qemu-devel@nongnu.org/msg34634.html
  • Sent to mentor the RFC for the new QMP command query-snapshots before sending to qemu-devel. This might avoid some noise in the list.
  • Lesson learned: compile qemu without --enable-debug before submitting anything, damn it! The do_savevm() v3 patch had an warning that was not caught by me and latter on caught by Kevin Wolf: http://www.mail-archive.com/qemu-devel@nongnu.org/msg34707.html After a few minutes chatting with him on IRC we figured that --enable-debug disables -O2, then some warnings could not be seen. Fixed and resent v4. I hope that know this think gets merged :-D

June 3

May 31 thru June 2

  • Studied and toke some exams. Classes are over :-D

May 29

May 28

  • Some chat with mentor.

May 27

  • Got mentor's review about last do_info_netdevices(). Will begin fixing things after finishing some more do_savevm/do_loadvm work.
  • do_info_snapshots() now displays snapshot information of all devices.
qemu -hda ../a.qcow2 -hdb ../b.qcow2
QEMU 0.12.50 monitor - type 'help' for more information
(qemu) info snapshots 
Snapshot list from ide0-hd0:
ID   TAG       VM SIZE                DATE       VM CLOCK                                 UUID                          PARENT UUID
1                 1.5M 2010-05-27 12:24:45   00:00:05.640 f89787ea-c39e-4531-90cc-19d73abbc50e 00000000-0000-0000-0000-000000000000
2                 1.5M 2010-05-27 12:24:52   00:00:12.258 4fb1f028-febb-464a-8e61-8abdfde9adc8 f89787ea-c39e-4531-90cc-19d73abbc50e
3                  1.5M 2010-05-27 12:25:05   00:00:24.921 faeb873c-f3fd-40ab-97de-67c6f12fd5ff 4fb1f028-febb-464a-8e61-8abdfde9adc8

Snapshot list from ide0-hd1:
ID   TAG       VM SIZE                DATE       VM CLOCK                                 UUID                          PARENT UUID
1                    0 2010-05-27 11:33:11   00:00:02.781 8f5d24a0-9a92-4650-8b56-babaa7790700 00000000-0000-0000-0000-000000000000
2                    0 2010-05-27 11:53:39   00:00:12.073 9e471508-f9d9-46d6-815b-2c25b15152f3 00000000-0000-0000-0000-000000000000
3                    0 2010-05-27 11:53:40   00:00:12.867 c5a75fd0-2cea-433b-bd21-37025dff3cbd 9e471508-f9d9-46d6-815b-2c25b15152f3

May 26

  • Got parent snapshot saving working. Using global variable, not sure if it is best option.
  • Updated bdrv_snapshot_dump() to properly show the UUID information. It looks really big.
QEMU 0.12.50 monitor - type 'help' for more information
(qemu) info snapshots 
Snapshot devices: ide0-hd0
Snapshot list (from ide0-hd0):
ID   TAG       VM SIZE                DATE       VM CLOCK                                 UUID                          PARENT UUID
1                 1.5M 2010-05-26 21:51:02   00:00:03.263 ce7cb390-f5d8-4633-9c83-e2c651d5aecf 00000000-0000-0000-0000-000000000000
2                 1.5M 2010-05-26 21:51:09   00:00:08.844 1f80df59-470e-4de6-a0a1-cf6a2a721e0e ce7cb390-f5d8-4633-9c83-e2c651d5aecf
3                 1.5M 2010-05-26 21:51:24   00:00:23.274 63ce278d-26fe-4790-ab13-6643bf9ddf90 1f80df59-470e-4de6-a0a1-cf6a2a721e0e
4                 1.5M 2010-05-26 21:53:17   00:00:03.595 0199cdd9-1c8f-4c53-b711-684110078960 00000000-0000-0000-0000-000000000000
5                 1.5M 2010-05-26 21:53:25   00:00:10.750 6b753c35-8939-477b-a7f5-10ff45e34e93 0199cdd9-1c8f-4c53-b711-684110078960
6                 1.5M 2010-05-26 21:53:27   00:00:12.071 7a127832-7259-4b41-9400-64804b755b0c 6b753c35-8939-477b-a7f5-10ff45e34e93

  • Fixed extra_data_size offset handling when loading a qcow2 image with an empty extra area.

May 22

I was actually not saving anything. After closing the VM the UUID information was not written to disk. After hours pulling some hair I figured it as a typo and a wrong offset calculation when saving the QCowSnapshotHeader to disk.

May 21

  • Got saving UUID information in a qcow2 file, using QCowSnapshot->extra_data_size and displaying it on 'info snapshots'. It seams we can have snapshot inheritance without breaking the file format or any old code. Now time to actually save the inheritance.

Helpful references:

May 19

After talking with Chris Lalancette from libvirt, we need snapshot inheritance in qemu, before converting do_savevm/do_loadvm to QMP. I've analyzed the code and I there is some hairy things to do. There are more people working on this code, so I will try to get a go before starting coding.

From late March thru May 18

Sent patch v4 to qemu-devel.

We were struggling to get the output of 'info network' exactly the same and juggling to work around the lack of a pattern when formatting VLANClientState.info_str. So we decided to add a new and clean command, and leave 'info network' as it is.

This is the v4 patch:

http://lists.gnu.org/archive/html/qemu-devel/2010-05/msg01567.html

The work on this area has begun late March, as a test if I could do something in GSoC. And here I am :-D There were a lot of interactions before the v4 patch.

Previous attempts: