I m trying to adopt ignite to resolve one of my needs by implementing an in memory data grid
Right know I m using a 3rd persistence read/write through mechanism to fetch data from my oracle db, in my topology I m using a single node in my activated cluster that are hosted in a VM having 8G ram and 120 G hdd.
my node is using local storage with a on-heap memory of 2G and 50G off-heap memory, all this with eviction and swapping enabled
DataStorageConfiguration dataStorageCfg = new DataStorageConfiguration();
DataRegionConfiguration dataRegionCfg = new DataRegionConfiguration();
// 2G initial size (RAM).
dataRegionCfg.setInitialSize(2L * 1024 * 1024 * 1024);
// 40 GB max size (RAM).
dataRegionCfg.setMaxSize(40L * 1024 * 1024 * 1024);
// Enabling RANDOM_LRU eviction for this region.
dataRegionCfg.setPageEvictionMode(DataPageEvictionMode.RANDOM_LRU);
//dataRegionCfg.setPersistenceEnabled(true);
final String swapPath ="/opt/ignite/swap";
dataRegionCfg.setSwapPath(swapPath);
dataStorageCfg.setDefaultDataRegionConfiguration(dataRegionCfg);
cfg.setDataStorageConfiguration(dataStorageCfg);
Caching this on my machine is taking too much time
my swap folder is about 13G when caching comes to end concerning my SQL query there is no response
the same query in my tools takes 1min45s to respond but using ignite cache query method doesn't respond and doesn't trow any kind of error or exception
SqlFieldsQuery sqlQuery;
FieldsQueryCursor<List<?>> queryCursor;
Iterator<List<?>> resultIt;
System.out.println(">>> All caches loaded! in : " + total + " ms");
System.out.println("---------------------------------------------- ");
System.out.println("---------------------------------------------- ");
System.out.println("---------------------------------------------- ");
System.out.println("\\n \\n \\n ");
System.out.println("---------------------------------------------- ");
System.out.println("---------------------------------------------- ");
System.out.println("---------------------------------------------- ");
System.out.println("Checking join query POC first run");
start = System.currentTimeMillis();
sqlQuery = new SqlFieldsQuery(sql);
queryCursor = ignite.cache("MInoutlineCache").query(sqlQuery);
System.out.println("query result size is : "+queryCursor.getAll().size());
end = System.currentTimeMillis() - start;
total += end;
M I well using ignite? is using ignite in a one node fashion useful, or I was meant to build clusters with lot of nodes in a partitioned strategy ?
the number of lines cached is 10 million line is there another way to achieve a good in memory data-grid in my context using the 3rd persistence strategy.
There is lot of question in this question I m sorry
NB: I m using the grid gain console to generate my configuration I m also updating my caching schema names as public to execute query directly
Here is the query
SELECT bp.name,
CF.documentno,
CF.MOVEMENTDATE,
CF.m_product_id AS M_PRODUCT_ID,
CF.product,
CF.xx_lignegratuite,
CF.m_attributesetinstance_id,
CASE
WHEN cf.isreturntrx='Y'
THEN - CF.qtyentered
ELSE CF.qtyentered
END AS qtyentered,
CF.discount,
CF.DOCSTATUS,
CF.ISRETURNTRX,
CF.XX_REWARDAMT,
CF.OperID,
CF.clientId,
CASE
WHEN cf.xx_lignegratuite='N'
THEN
CASE
WHEN cf.isreturntrx='Y'
THEN -cf.prixVente*cf.qtyentered* (1-(cf.discount/100))
ELSE cf.prixVente*cf.qtyentered* (1-(cf.discount/100))
END
ELSE 0
END AS totalline,
CASE
WHEN cf.XX_StartegicalProduct='Y'
THEN (
CASE
WHEN cf.xx_lignegratuite='N'
THEN
CASE
WHEN cf.isreturntrx='Y'
THEN -cf.prixVente*cf.qtyentered* (1-(cf.discount/100))
ELSE cf.prixVente*cf.qtyentered* (1-(cf.discount/100))
END
ELSE 0
END)
ELSE 0
END AS totallineStar,
CASE
WHEN cf.xx_lignegratuite='N'
THEN
CASE
WHEN cf.typevente='W'
THEN
CASE
WHEN cf.isreturntrx='Y'
THEN -(cf.XX_REWARDAMT/nb_doc)
ELSE cf.XX_REWARDAMT/nb_doc
END
ELSE 0
END
ELSE 0
END AS totalreward,
CF.XX_StartegicalProduct,
CF.SALESREP_ID,
CF.C_DOCTYPE_ID,
CF.AD_ORG_ID,
CF.ad_orgtrx_id,
CF.xx_laboratory_id,
bp.c_bpartner_id,
CF.nb_doc,
CF.rate,
CF.poste_id,
CF.SalesRepTier_poste_id,
CF.recSupr,
CF.recSupr_poste_id,
(SELECT Objectif_CA_oper
FROM c_bpartner
WHERE issalesrep ='Y'
AND isemployee ='Y'
AND c_bpartner_id=CF.SalesRepTier
) AS ObjectifOp,
(SELECT Objectif_CA_oper
FROM c_bpartner
WHERE issalesrep ='Y'
AND isemployee ='Y'
AND c_bpartner_id=CF.SalesRepTier_poste_id
) AS ObjectifOp_poste_id,
CASE
WHEN cf.ISRETURNTRX='Y'
THEN -cf.QTYENTERED*prixRevient
ELSE cf.QTYENTERED*prixRevient
END AS consomation
FROM
(SELECT i.documentno,
i.MOVEMENTDATE,
p.m_product_id,
p.name AS product,
ol.xx_lignegratuite,
il.m_attributesetinstance_id,
il.qtyentered,
ol.discount,
i.DOCSTATUS,
i.isreturntrx,
ol.XX_REWARDAMT,
i.C_BPartner_ID AS clientId,
(SELECT u.C_BPARTNER_ID FROM AD_User u WHERE u.AD_User_ID = i.SALESREP_ID
) AS OperID,
(SELECT ai.Valuenumber
FROM M_AttributeInstance ai
INNER JOIN M_Attribute a
ON (ai.M_Attribute_ID =a.M_Attribute_ID
AND a.IsInstanceAttribute ='Y')
WHERE ai.M_AttributeSetInstance_ID=il.m_attributesetinstance_id
AND a.Name ='Prix Vente'
) AS prixVente,
(SELECT ai.Valuenumber
FROM M_AttributeInstance ai
INNER JOIN M_Attribute a
ON (ai.M_Attribute_ID =a.M_Attribute_ID
AND a.IsInstanceAttribute ='Y')
WHERE ai.M_AttributeSetInstance_ID=il.m_attributesetinstance_id
AND a.Name ='Prix Revient'
) AS prixRevient,
(SELECT ai.Valuenumber
FROM M_AttributeInstance ai
INNER JOIN M_Attribute a
ON (ai.M_Attribute_ID =a.M_Attribute_ID
AND a.IsInstanceAttribute ='Y')
WHERE ai.M_AttributeSetInstance_ID=il.m_attributesetinstance_id
AND a.Name ='Fournisseur'
) AS Fournisseur,
XX_StartegicalProduct,
i.SALESREP_ID,
i.C_DOCTYPE_ID,
i.AD_ORG_ID,
(SELECT o.AD_ORGTRX_ID
FROM c_order o
WHERE i.c_order_id=o.c_order_id
) AS ad_orgtrx_id,
p.xx_laboratory_id,
lt.rate,
--COUNT(*) over (partition BY il.c_orderline_id) AS nb_doc,
(
SELECT COUNT(*)
FROM m_inoutline ill
WHERE ill.c_orderline_id=il.c_orderline_id
) AS nb_doc,
ol.type AS typevente,
bpl.salesrep_id AS poste_id,
(SELECT u.c_bpartner_id FROM AD_User u WHERE u.AD_User_ID = i.salesrep_id
) AS SalesRepTier,
(SELECT u.c_bpartner_id FROM AD_User u WHERE u.AD_User_ID = bpl.salesrep_id
) AS SalesRepTier_poste_id,
(SELECT u.XX_RecSupervisor_ID FROM AD_User u WHERE u.AD_User_ID=i.SALESREP_ID
) AS recSupr,
(SELECT u.XX_RecSupervisor_ID
FROM AD_User u
WHERE u.AD_User_ID=bpl.salesrep_id
) AS recSupr_poste_id
FROM m_inoutline il
INNER JOIN m_inout i
ON il.m_inout_id=i.m_inout_id
INNER JOIN c_orderline ol
ON ol.c_orderline_id=il.c_orderline_id
INNER JOIN m_product p
ON p.m_product_id=il.m_product_id
INNER JOIN C_Bpartner bpl
ON (bpl.c_bpartner_id=i.c_bpartner_id)
LEFT OUTER JOIN xx_listetauxvaleur lt
ON p.xx_listetauxvaleur_id = lt.xx_listetauxvaleur_id
WHERE i.issotrx ='Y'
--AND p.m_attributeset_id IS NOT NULL
AND il.movementqty<>0
) CF
LEFT OUTER JOIN c_bpartner bp
ON (CF.Fournisseur=bp.c_bpartner_id)
ORDER BY bp.name,
documentno;
I see a lot of joins in your statement. Are you sure that you have indexes on all the fields present in those joins? I think it would take a lot of optimization until this one will perform.
I also recommend getting rid of swap since this is an outdated feature and its performance implications are not known.
If you expect to outperform Oracle on a single node when doing multiple joins, this is also not going to happen.