Source control and ip cores

[Ilya Kalistru]

I'm looking for a peace of advice.
Currently I use git for version control in my VHDL projects and I usually include all .vhd files as well as .xdc constrains files in it. But I don't know what I should do with ip cores. Including the whole directory of ip core to repository doesn't seem to be a good idea because there are to many files and too many of them are changed after any change of ip settings. Other choice, manual description of ip settings in comments is tedious and error prone.

 

[Brian Drummond]

On Sun, 07 Feb 2016 00:11:17 -0800, Ilya Kalistru wrote:

I'm looking for a peace of advice.
Currently I use git for version control in my VHDL projects and I
usually include all .vhd files as well as .xdc constrains files in it.
But I don't know what I should do with ip cores. Including the whole
directory of ip core to repository doesn't seem to be a good idea
because there are to many files and too many of them are changed after
any change of ip settings. Other choice, manual description of ip
settings in comments is tedious and error prone.

I hear you loud and clear. FPGA tool vendors are a couple of decades
behind in their understanding of development practices, and often don't
seem to consider version control at all.

It takes some digging and experimentation to work out which
configuration files you need to regenerate the IP cores from, and how to
regenerate them with minimal GUI interaction.

Maybe worth digging around in the "command log" files to see which
command line tools are run when you press "Generate", and which files
they take as inputs. That command line can then be extracted for
scripting the process.

Specific details will depend on the tools of course.

-- Brian

 

[Ilya Kalistru]

On Sunday, February 7, 2016 at 3:15:25 PM UTC+3, Brian Drummond wrote:

On Sun, 07 Feb 2016 00:11:17 -0800, Ilya Kalistru wrote:

I'm looking for a peace of advice.
Currently I use git for version control in my VHDL projects and I
usually include all .vhd files as well as .xdc constrains files in it.
But I don't know what I should do with ip cores. Including the whole
directory of ip core to repository doesn't seem to be a good idea
because there are to many files and too many of them are changed after
any change of ip settings. Other choice, manual description of ip
settings in comments is tedious and error prone.

I hear you loud and clear. FPGA tool vendors are a couple of decades
behind in their understanding of development practices, and often don't
seem to consider version control at all.

It takes some digging and experimentation to work out which
configuration files you need to regenerate the IP cores from, and how to
regenerate them with minimal GUI interaction.

Maybe worth digging around in the "command log" files to see which
command line tools are run when you press "Generate", and which files
they take as inputs. That command line can then be extracted for
scripting the process.

Specific details will depend on the tools of course.

-- Brian

Ok. That is the approach I'm going to try. It has one disadvantage - configuration files are not really human-readable (xilinx .xci), but it looks like the best way by now.
Thank you for response.

 

[Mark Curry]

In article <n97cbb$re0$2@dont-email.me>,
Brian Drummond <brian@shapes.demon.co.uk> wrote:

On Sun, 07 Feb 2016 00:11:17 -0800, Ilya Kalistru wrote:

I'm looking for a peace of advice.
Currently I use git for version control in my VHDL projects and I
usually include all .vhd files as well as .xdc constrains files in it.
But I don't know what I should do with ip cores. Including the whole
directory of ip core to repository doesn't seem to be a good idea
because there are to many files and too many of them are changed after
any change of ip settings. Other choice, manual description of ip
settings in comments is tedious and error prone.

I hear you loud and clear. FPGA tool vendors are a couple of decades
behind in their understanding of development practices, and often don't
seem to consider version control at all.

Preach it Brian. Version control (at least for vendor X) is a late
addition, very poorly bolted on. The design flows were created with
an image of a single user owning and maintaining a single FPGA by
themselves. Very 90's.

It takes some digging and experimentation to work out which
configuration files you need to regenerate the IP cores from, and how to
regenerate them with minimal GUI interaction.

This. We'll often check in a zip'ed archive of the dummy run we used to create
the IP - along with ALL it's junk. Just for information purposes.
As to what's actually USED in FPGA development, we usually peel back all the
layers of cruft, and find the source RTL under all the chaff. That's
what's actually used in our scripts / development.

It's usually much more flexible to use the actual RTL as well, as opposed to
the crap they bolt on top of it.

Regards,

Mark

 

[Theo Markettos]

Mark Curry <gtwrek@sonic.net> wrote:

In article <n97cbb$re0$2@dont-email.me>,
Brian Drummond <brian@shapes.demon.co.uk> wrote:

I hear you loud and clear. FPGA tool vendors are a couple of decades
behind in their understanding of development practices, and often don't
seem to consider version control at all.

I'm also quite tired of vendors trying to re-implement their own version
control systems. Altium has 'vaults', for instance, which are essentially a
poor interface to Subversion ten years too late. The files are still binary
and there's no usable merging. That's no good when your vcs of choice is
git/bzr/hg/...

This. We'll often check in a zip'ed archive of the dummy run we used to
create the IP - along with ALL it's junk. Just for information purposes.
As to what's actually USED in FPGA development, we usually peel back all
the layers of cruft, and find the source RTL under all the chaff. That's
what's actually used in our scripts / development.

It's usually much more flexible to use the actual RTL as well, as opposed
to the crap they bolt on top of it.

We have the opposite problem. As an open source project we cannot ship the
vendor RTL, because it usually says (c) Vendor, Do Not Distribute or
something on it. But checking in the vendor RTL is the only way to get
things done.

So we have a split workflow: devs check in whatever vendor cruft into the
private repo they need to make things work. Then, when we want to cut a
public release, we have to go through and work out how to generate the RTL
from licence-clean sources by some invocation of the tools. Once the
invocations are put in the makefile, we can then build and check things
still work with the generated RTL. Unfortunately the tools are often not
friendly to this workflow.

Theo

 

[Daniel Kho]

On Sunday, 7 February 2016 21:50:30 UTC+8, Ilya Kalistru wrote:

On Sunday, February 7, 2016 at 3:15:25 PM UTC+3, Brian Drummond wrote:
On Sun, 07 Feb 2016 00:11:17 -0800, Ilya Kalistru wrote:

I'm looking for a peace of advice.
Currently I use git for version control in my VHDL projects and I
usually include all .vhd files as well as .xdc constrains files in it..
But I don't know what I should do with ip cores. Including the whole
directory of ip core to repository doesn't seem to be a good idea
because there are to many files and too many of them are changed after
any change of ip settings. Other choice, manual description of ip
settings in comments is tedious and error prone.

I hear you loud and clear. FPGA tool vendors are a couple of decades
behind in their understanding of development practices, and often don't
seem to consider version control at all.

It takes some digging and experimentation to work out which
configuration files you need to regenerate the IP cores from, and how to
regenerate them with minimal GUI interaction.

Maybe worth digging around in the "command log" files to see which
command line tools are run when you press "Generate", and which files
they take as inputs. That command line can then be extracted for
scripting the process.

Specific details will depend on the tools of course.

-- Brian

Ok. That is the approach I'm going to try. It has one disadvantage - configuration files are not really human-readable (xilinx .xci), but it looks like the best way by now.
Thank you for response.

Xilinx's configuration files aren't really that human-unreadable. I am actually amazed at how much standards support Xilinx puts into their tool and flows. If you take a good look into an *.xci file with a text editor, you will notice that it is in a human-readable XML format. And this is no ordinary XML - the format of the XML actually adheres to the IP-XACT specification, which is indeed a portable method to package and transfer components between tools that support this specification.
https://en.wikipedia.org/wiki/IP-XACT

-daniel

 

[Daniel Kho]

On Tuesday, 23 February 2016 20:38:17 UTC+8, Daniel Kho wrote:

On Sunday, 7 February 2016 21:50:30 UTC+8, Ilya Kalistru wrote:
On Sunday, February 7, 2016 at 3:15:25 PM UTC+3, Brian Drummond wrote:
On Sun, 07 Feb 2016 00:11:17 -0800, Ilya Kalistru wrote:

I'm looking for a peace of advice.
Currently I use git for version control in my VHDL projects and I
usually include all .vhd files as well as .xdc constrains files in it.
But I don't know what I should do with ip cores. Including the whole
directory of ip core to repository doesn't seem to be a good idea
because there are to many files and too many of them are changed after
any change of ip settings. Other choice, manual description of ip
settings in comments is tedious and error prone.

I hear you loud and clear. FPGA tool vendors are a couple of decades
behind in their understanding of development practices, and often don't
seem to consider version control at all.

It takes some digging and experimentation to work out which
configuration files you need to regenerate the IP cores from, and how to
regenerate them with minimal GUI interaction.

Maybe worth digging around in the "command log" files to see which
command line tools are run when you press "Generate", and which files
they take as inputs. That command line can then be extracted for
scripting the process.

Specific details will depend on the tools of course.

-- Brian

Ok. That is the approach I'm going to try. It has one disadvantage - configuration files are not really human-readable (xilinx .xci), but it looks like the best way by now.
Thank you for response.

Xilinx's configuration files aren't really that human-unreadable. I am actually amazed at how much standards support Xilinx puts into their tool and flows. If you take a good look into an *.xci file with a text editor, you will notice that it is in a human-readable XML format. And this is no ordinary XML - the format of the XML actually adheres to the IP-XACT specification, which is indeed a portable method to package and transfer components between tools that support this specification.
https://en.wikipedia.org/wiki/IP-XACT

-daniel

Yes, I think it is a good idea to version control IP-XACT files.

-daniel

 

[Svenn Are Bjerkem]

Ilya Kalistru <stebanoid@gmail.com> wrote:

I'm looking for a peace of advice.
Currently I use git for version control in my VHDL projects and I usually
include all .vhd files as well as .xdc constrains files in it. But I don't
know what I should do with ip cores. Including the whole directory of ip core
to repository doesn't seem to be a good idea because there are to many files
and too many of them are changed after any change of ip settings. Other
choice, manual description of ip settings in comments is tedious and error
prone.

I check in only the xci and xml files in the sources_1/ip structure.

When checking out a pristine tree from source control and run generate
bitstream for the first time, Vivado will go through the process of
regenerating all the products of the included IP.

This is GUI workflow.

--
svenn

 

[Mark Curry]

In article <4b7d68c4-7824-40d8-a114-b0c331b979a2@googlegroups.com>,
Daniel Kho <daniel.kho@gmail.com> wrote:

On Sunday, 7 February 2016 21:50:30 UTC+8, Ilya Kalistru wrote:
On Sunday, February 7, 2016 at 3:15:25 PM UTC+3, Brian Drummond wrote:
On Sun, 07 Feb 2016 00:11:17 -0800, Ilya Kalistru wrote:

I'm looking for a peace of advice.
Currently I use git for version control in my VHDL projects and I
usually include all .vhd files as well as .xdc constrains files in it.
But I don't know what I should do with ip cores. Including the whole
directory of ip core to repository doesn't seem to be a good idea
because there are to many files and too many of them are changed after
any change of ip settings. Other choice, manual description of ip
settings in comments is tedious and error prone.

I hear you loud and clear. FPGA tool vendors are a couple of decades
behind in their understanding of development practices, and often don't
seem to consider version control at all.

It takes some digging and experimentation to work out which
configuration files you need to regenerate the IP cores from, and how to
regenerate them with minimal GUI interaction.

Maybe worth digging around in the "command log" files to see which
command line tools are run when you press "Generate", and which files
they take as inputs. That command line can then be extracted for
scripting the process.

Specific details will depend on the tools of course.

-- Brian

Ok. That is the approach I'm going to try. It has one disadvantage - configuration
files are not really human-readable (xilinx .xci), but it looks like the best way
by now.

Xilinx's configuration files aren't really that human-unreadable. I am actually amazed at
how much standards support Xilinx puts into their tool and flows. If you take a good look
into an *.xci file with a text editor, you will notice that it is in a human-readable XML
format. And this is no ordinary XML - the format of the XML actually adheres to the
IP-XACT specification, which is indeed a portable method to package and transfer components
between tools that support this specification.
https://en.wikipedia.org/wiki/IP-XACT

(Getting ranty, and not helping the OP, but this is a sore spot for us...)

Xilinx support of standards is 2 steps forwards, 1.9 steps back. Sure they're using
more "industry standard" files. (SDC, TCL). But they're still burying the landscape
in a host of unneccesary, undocumented, randomly changing list of other ancillary
files.

I'm really not clear as to what IP-XACT is trying to solve, and who it's audience is.
But XCI files are a very unreliable way of managing IP. Sure it's XML - there's
tools available to manipulate them. But Xilinx doesn't document the
data within those files in any way. They're free to change it at any time.
Can a user reliably MERGE changes within those files from one development
tree, and another? Will the design still build 1, 2, 5, or 10 years
down the line using that undocumented src file?

Can you simulate an XCI file? Can you run lint on it? Can you modify it's contents?
What's configurable within them? In the end, it's just something that gets in the way
of what you really need to do those tasks - the RTL (or netlist as neccesary -
encrypted at worst case).

Regards,

Mark

 

I'm working at a big project with multiple developers and we also want to be able to share sources in a version controlled friendly.
The project mainly uses X... FPGAs and V... tools.

One of possible approaches was to use IP packager, however it seems that it is not possible, or at least discouraged to use IP cores with port defined as VHDL records (just imagine a few AXI ports as bits and bit vectors, and compare it to the clean design of IPbus with record based ports - https://svnweb.cern.ch/trac/cactus )

The another approach I've proposed is based on extension of the old good PRJ files.

The file may contain lines of following types:
- type library file
(Line defining the source file
Possible types: vhdl, verilog, system, header. xci, xdc
File name (path in fact) is defined relatively to the location of the
PRJ file)

- include file
(Line defining the include file
File name (path in fact) is defined relatively to the location of the
PRJ file
Use of relative paths allows easy reuse of so defined
blocks. Use of simple text files makes such solution
SVN/GIT friendly

 

Oooops, sorry, I've unintentionally clicked "publish" before my message was ready :-(
Here it is again in a complete form.


I'm working at a big project with multiple developers and we also want to be able to share sources in a version controlled friendly.
The project mainly uses X... FPGAs and V... tools.

One of possible approaches was to use IP packager, however it seems that it is not possible, or at least discouraged to use IP cores with port defined as VHDL records (just imagine a few AXI ports as bits and bit vectors, and compare it to the clean design of IPbus with record based ports - https://svnweb.cern.ch/trac/cactus )

The another approach I've proposed is based on extension of the old good PRJ files.

The file may contain lines of following types:
- type library file
(Line defining the source file
Possible types: vhdl, verilog, system, header. xci, xdc
File name (path in fact) is defined relatively to the location of the
PRJ file)

- include file
(Line defining the include file
File name (path in fact) is defined relatively to the location
of the PRJ file)

Use of relative paths allows easy reuse of so defined
blocks. Use of simple text files makes such solution
SVN/GIT friendly

I've prepared a simple routine, which translates PRJ file (with included
files) to the list of files, based on the "prj_add" routine
from UG946 tutorial.

Here it is:

# Procedure below reads the source files from PRJ files, extended with
# the "include file" statement

#Important thing - path to the source files should be given relatively
#to the location of the PRJ file.
proc read_prj { prj } {
#initialize results to an empty list
set res []
if {[file exists $prj]} {
puts "\tReading PRJ file: $prj"
set source [open $prj r]
set source_data [read $source]
close $source
#Extract the directory of the PRJ file, as all paths to the
#source files must be given relatively to that directory
set prj_dir [ file dirname $prj ]
regsub -all {\"} $source_data {} source_data
set prj_lines [split $source_data "\n" ]
set line_count 0
foreach line $prj_lines {
incr line_count
#Ignore empty and commented lines
if {[llength $line] > 0 && ![string match -nocase "#*" $line]} {
#Detect the inlude line
lassign $line type file
if {[string match -nocase $type "include"]} {
puts "\tIncluding PRJ file: $prj_dir/$file"
set inc [ read_prj $prj_dir/$file ]
foreach l $inc {
lappend res $l
}
} else {
lappend res [linsert $line 0 $prj_dir]
}
}
}
}
return $res
}

Of course the above procedure is a quick&dirty solution lacking the full error detection.

Anyway it is possible to define each IP block as a tree of directories containing sources and associated extended PRJ files.

The main disadvantage is that this approach is not supported by the tool vendor, so it is unclear whether it will work with the future versions...

Maybe this idea may be somehow improved?

With best regards,
Wojtek

 

Yet one thing which was missed in my previous post.
How the read_prj procedure is used in the Tcl script building the project?
Below is the appropriate part of the Tcl script:

set prj src/top.prj
set prj_lines [ read_prj $prj ]
foreach line $prj_lines {
# Just read the type
puts $line
lassign $line pdir type lib file
# Handle the source files
if { \
[string match -nocase $type "xci"] || \
[string match -nocase $type "header"] || \
[string match -nocase $type "system"] || \
[string match -nocase $type "verilog"] || \
[string match -nocase $type "vhdl"]} {

set nfile [file normalize "$pdir/$file"]
add_files -norecurse -fileset $sobj $nfile
set file_obj [get_files -of_objects $sobj $nfile]
#Handle VHDL file
if {[string match -nocase $type "vhdl"]} {
set_property "file_type" "VHDL" $file_obj
set_property "library" $lib $file_obj
}
#Handle Verilog file
if {[string match -nocase $type "verilog"]} {
set_property "file_type" "Verilog" $file_obj
set_property "library" $lib $file_obj
}
#Handle SystemVerilog file
if {[string match -nocase $type "system"]} {
set_property "file_type" "SystemVerilog" $file_obj
}
#Handle Verilog header file
if {[string match -nocase $type "header"]} {
set_property "file_type" "Verilog Header" $file_obj
}
#Handle XCI file
if {[string match -nocase $type "xci"]} {
#set_property "synth_checkpoint_mode" "Singular" $file_obj
}
}
if { [string match -nocase $type "xdc"]} {
set nfile [file normalize "$pdir/$file"]
add_files -norecurse -fileset $cobj $nfile
set file_obj [get_files -of_objects $cobj $nfile]
set_property "file_type" "XDC" $file_obj
}
}

With best regards,
Wojtek

 

[Ilya Kalistru]

On Tuesday, February 23, 2016 at 3:38:17 PM UTC+3, Daniel Kho wrote:

> Xilinx's configuration files aren't really that human-unreadable.

I meant that it contains huge amount of parameters and sometimes it's unclear what they mean.

 

I have published the usable version of my scripts for complex sources hierarchy management on the github: https://github.com/wzab/vextproj

A copy of current version is also available in alt.sources:
https://groups.google.com/forum/#!original/alt.sources/nPIluidTlMU/bv-oEzMrHgAJ

With best regards,
Wojtek

 

Hi,
I have added support for "Block Design" components to my vextproj project.
There is also a simple demo added, running on the Z-Turn board (http://www.myirtech.com/list.asp?id=502).
The demo compiles the bitstream from the project built using the EPRJ files.
Additionally the demo allows you to compile Linux with the appropriate device driver and test program, using the Buildroot environment.

With best regards,
Wojtek

 

W dniu środa, 4 maja 2016 18:55:44 UTC+2 użytkownik wza...@gmail.com napisał:

Hi,
I have added support for "Block Design" components to my vextproj project..
There is also a simple demo added, running on the Z-Turn board (http://www.myirtech.com/list.asp?id=502).
The demo compiles the bitstream from the project built using the EPRJ files.
Additionally the demo allows you to compile Linux with the appropriate device driver and test program, using the Buildroot environment.

With best regards,
Wojtek

Of course the project is still at https://github.com/wzab/vextproj